Agentic AI environments like OpenClaw execute arbitrary code with tool access – they can read files, spawn processes, and make network requests. Running such workloads on a local machine or an unsandboxed server is inherently unsafe:

Agents inherit host-level privileges and can access the entire filesystem, including credentials and private keys
Without network policy enforcement, agents can exfiltrate data or reach arbitrary endpoints
A misbehaving agent on a local machine has no isolation boundary; the blast radius is everything on the host
Containers without network controls only solve half the problem – an agent with unrestricted egress can still leak data

KubeClaw provides a fully automated Kubernetes cluster on Hetzner Cloud VPS servers where OpenClaw runs inside containers with strict network controls. Infrastructure is managed through OpenTofu and Ansible, the cluster uses Cilium CNI for eBPF-based network policies that enforce per-namespace egress rules (e.g., allowing only Anthropic API and messaging provider endpoints), and all access is routed through a Cloudflare Tunnel – no open ports, no public SSH, outbound-only connectivity.

How it works

OpenTofu provisions the infrastructure: private network, firewalls, SSH keys, and servers on Hetzner Cloud
Cloud-init configures each server on first boot: SSH hardening, fail2ban, UFW, NAT64/DNS64, and Kubernetes prerequisites
Ansible handles ongoing server management: updates, security hardening, and configuration changes
kubeadm bootstraps a standard Kubernetes cluster with Cilium CNI and Hetzner CSI for persistent storage
Cloudflare Tunnel provides secure, outbound-only SSH access without exposing any ports
Cilium network policies enforce per-namespace egress rules, restricting OpenClaw to only its required API endpoints

For the node roles, IP layout, and traffic flow behind this, see Architecture.

Features

Container isolation – Agentic workloads run in Kubernetes pods, never on bare metal or your local machine
Cilium network policies – eBPF-based FQDN egress filtering limits what an agent can reach on the network
Cloudflare Tunnel – Secure SSH and service access without open ports; outbound-only connectivity
IPv6-only – No public IPv4 addresses required, NAT64/DNS64 for transparent IPv4 reachability
Admin Node – Temporary jump host with public IPv6 for initial setup (removable)
Scalable – Master + replica control nodes, 0 to n worker nodes, mixed server types
Custom SSH keys – Optionally use your own keys, with configurable key file prefix
Ansible-ready – Playbooks for updates, hardening, NAT64 configuration, and Kubernetes prerequisites
Debian 13 – Stable, Kubernetes-compatible OS
kubeadm – Standard Kubernetes bootstrapper for CKA certification preparation

Where to start

If you want to…	Go to
Deploy a cluster now	Quick Start
Understand the design first	Introduction – architecture, security model, DNS/NAT64
Follow the full deployment path	Guide – Dev Container through OpenClaw, in order
Solve one specific task	How-to
Look up a variable, output, or template	Reference
Run the cluster day to day	Operations

1 - Project Status

What KubeClaw is, what it is not, and what you should not do with it.

KubeClaw is a prototype

KubeClaw is a learning project. Its objective is to explore what a secure, scalable environment for running AI agents on Kubernetes actually requires – by building one end to end and finding out where the difficulties are.

It is not production ready. It is not beta. There is no supported release, no stability guarantee, and no security review.

What that means concretely

	Status
Maturity	Prototype / experiment
Suitable for production	No
Suitable for handling real secrets or customer data	No
Security reviewed or audited	No
API / variable stability	None – variables and layouts change without notice
Support	None – issues may go unanswered
Breaking changes	Expected, without a deprecation period

Why it exists

Agentic AI workloads execute arbitrary code with tool access. Running them without an isolation boundary is genuinely risky, and the interesting question is what a correct boundary looks like: which network controls actually hold, how much egress restriction is practical, what the operational cost is, and where the design breaks under load.

KubeClaw is an attempt to answer those questions by construction. It combines IPv6-only Hetzner infrastructure, OpenTofu and Ansible provisioning, kubeadm, Cilium network policies with FQDN egress filtering, and Cloudflare Tunnel access. Building the whole path surfaces problems that reading about it does not.

A secondary objective is preparation for the CKA certification, which is why the cluster uses kubeadm rather than a turnkey distribution.

What it is not

Not a product. There is no roadmap commitment and no support channel.
Not a reference architecture. Several decisions are made to be instructive rather than optimal.
Not hardened. The security model documented here describes the intent of the design. It has not been adversarially tested, and you should assume gaps exist.
Not a template to fork for production. More mature approaches are expected to follow in separate projects, informed by what this one gets wrong.

If you are evaluating this for real work

Please do not deploy KubeClaw to handle production traffic, real credentials, or third-party data. Read it, take the ideas, and build something you have reviewed yourself. The infrastructure it provisions is real and costs real money -- see [Cost Estimate](/kubeclaw/docs/reference/cost-estimate/) before running `tofu apply`.

Current state

The infrastructure layer is functional: network, firewalls, SSH key handling, cloud-init, and the admin/control/worker node roles all provision and come up. NAT64/DNS64, the Ansible playbooks, and the documentation site are in place.

The Kubernetes and OpenClaw layers are documented as guides but are not yet automated, and the Kubernetes-specific firewall rules are deliberately still excluded from the OpenTofu configuration. See the Roadmap for what comes next.

Versioning

Releases use v0.x and follow semantic versioning only loosely. While the major version is 0, any release may break any interface. See the changelog for what changed.

2 - Quick Start

Provision the cluster end to end: prerequisites, Dev Container, OpenTofu, SSH, and the Cloudflare Tunnel.

This provisions real, billable infrastructure

KubeClaw is a [prototype](/kubeclaw/docs/project-status/) -- a learning project, not production software. The steps below create servers on Hetzner Cloud that cost real money, and the resulting cluster has not been security reviewed. Do not use it for production traffic, real credentials, or third-party data.

Prerequisites

Accounts

Hetzner Cloud account with an API token (console.hetzner.cloud)
Cloudflare account with a configured domain (free tier is sufficient)

Local Machine

Docker and an IDE with Dev Container support (e.g. VS Code + Dev Containers extension)

That’s it. All project tools (OpenTofu, Ansible, SSH, cloudflared, Hugo, Kubernetes clients, and AI assistants) are available in the Dev Container. No local installation is required beyond Docker and an IDE with Dev Container support.

Optional

cloudflared on your local machine for SSH via Cloudflare Tunnel (brew install cloudflared on macOS). It is included in the Dev Container, but is useful on the host if you connect outside the container.
Dashlane or another password manager for storing SSH keys and API tokens securely.

1. Clone and prepare

git clone <repo-url>
cd kubeclaw

# Create the persistent SSH directory used by the aibox Dev Container.
mkdir -p .aibox-home/.ssh
chmod 700 .aibox-home/.ssh

2. Open in Dev Container

Open the project in your IDE and start the Dev Container (e.g. VS Code: “Reopen in Container”).

All remaining commands run inside the Dev Container.

3. Configure

cp terraform.tfvars.example terraform.tfvars
# Edit terraform.tfvars with your Hetzner API token and other settings

See Variables Reference for all available configuration options.

4. Create infrastructure

tofu init
tofu apply

This creates the admin node (temporary jump host), control node, private network, and firewalls. The admin node is enabled by default for initial SSH setup.

5. Set up SSH

./scripts/setup-ssh.sh
source ./scripts/ssh-agent-setup.sh
./scripts/generate-ansible-inventory.sh

This exports SSH keys from the OpenTofu state, generates ~/.ssh/config entries, starts the ssh-agent with the cluster keys loaded, and builds ansible/inventory.ini from the current state. The project override makes .aibox-home/.ssh/ writable in the container, so these files persist across container rebuilds without being committed.

Rerun after every infrastructure change

`generate-ansible-inventory.sh` reads the OpenTofu outputs. Run it again after any `tofu apply` that adds or removes nodes -- never hand-edit `ansible/inventory.ini`.

6. Connect to the control node

ssh control-node   # Routes via admin node automatically

7. Install Cloudflare Tunnel

Creating the tunnel, routing SSH through it, and adding an Access policy are covered step by step in the Cloudflare Tunnel Setup guide. Once you have a tunnel token, install it one of two ways.

Option A: Automatic (recommended) – Set the tunnel token in terraform.tfvars before tofu apply:

cloudflare_tunnel_token = "eyJ..."

Get the token from the Cloudflare Zero Trust Dashboard under Networks > Tunnels > Create/Configure. The tunnel auto-starts on boot and survives node recreation.

Option B: Manual – SSH to the control node and install manually:

sudo cloudflared service install <YOUR_TUNNEL_TOKEN>

8. Disable admin node

After the tunnel is working, disable the temporary admin node:

# terraform.tfvars
enable_admin_node = false

tofu apply

The master control node always keeps public IPv6 (required for cloudflared).

Next steps

Server management with Ansible – updates, hardening, and Kubernetes prerequisites
Deploy Kubernetes with kubeadm
Deploy OpenClaw into an egress-restricted namespace
DNS and NAT64 for IPv4 reachability

3 - Introduction

Understand KubeClaw’s architecture, network model, and security boundaries.

Start here to understand the design decisions behind KubeClaw before provisioning infrastructure.

3.1 - Architecture

Node roles, private network layout, and how traffic reaches an IPv6-only cluster.

Overview

KubeClaw creates a secure, IPv6-only Kubernetes cluster on Hetzner Cloud. The design prioritizes security through network isolation: no public IPv4 addresses, SSH access exclusively via Cloudflare Tunnel, and per-namespace egress control with Cilium network policies.

┌─────────────────────────────────────────────────────────────────┐
│                         Internet                                 │
└─────────────────────────────────────────────────────────────────┘
                         │              │
                         ▼              ▼
              ┌──────────────┐  ┌───────────────┐
              │  Cloudflare  │  │  Admin Node   │
              │   Tunnel     │  │  10.0.0.254   │
              │  (permanent) │  │  (temporary,  │
              └──────┬───────┘  │  public IPv6) │
                     │          └───────┬───────┘
                     ▼                  ▼
┌─────────────────────────────────────────────────────────────────┐
│                    Hetzner Cloud                                 │
│  ┌─────────────────────────────────────────────────────────────┐ │
│  │              Private Network (10.0.0.0/24)                  │ │
│  │                                                             │ │
│  │   ┌─────────────────┐  ┌─────────────────┐                 │ │
│  │   │  control-01     │  │  control-02+    │                 │ │
│  │   │    10.0.0.2     │  │   10.0.0.3+     │                 │ │
│  │   │  (master,       │◄►│  (replicas,     │                 │ │
│  │   │   cloudflared)  │  │   0-n instances) │                 │ │
│  │   └────────┬────────┘  └─────────────────┘                 │ │
│  │            │                                                │ │
│  │            ▼                                                │ │
│  │   ┌─────────────────┐                                       │ │
│  │   │  worker-nodes   │                                       │ │
│  │   │  (0-n instances) │                                       │ │
│  │   └─────────────────┘                                       │ │
│  │                                                             │ │
│  └─────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘

Node Roles

Master Control Node (control-01, 10.0.0.2)

The master control node always exists and serves as:

Kubernetes control plane – runs etcd, kube-apiserver, kube-scheduler, kube-controller-manager
Cloudflare Tunnel endpoint – runs cloudflared for SSH access from the internet
SSH gateway – all other nodes are reached through this node

The master node always has public IPv6 (required for cloudflared outbound connections) and accepts SSH from the private network and from localhost (for the Cloudflare Tunnel). The tunnel can be auto-configured via cloudflare_tunnel_token or installed manually.

Replica Control Nodes (control-02+, 10.0.0.3+)

Optional nodes for high-availability control plane. Same configuration as the master, minus cloudflared. IPs start at 10.0.0.3 and increment.

Worker Nodes (10.0.0.x, offset after replicas)

Optional compute nodes for running workloads. Workers have restricted connectivity:

Inbound: SSH from private network only
Outbound: DNS (port 53), HTTP (port 80), HTTPS (port 443), internal network only
No TCP forwarding – prevents workers from being used as jump hosts

Worker IPs start after the last replica control node.

Admin Node (10.0.0.254, temporary)

A temporary jump host with public IPv6, used only during initial setup before the Cloudflare Tunnel is configured. Created by default (enable_admin_node = true) and should be disabled after tunnel setup.

Network Design

Private Network (10.0.0.0/24)

All nodes communicate via a Hetzner private network. IP assignments:

Address	Node
`10.0.0.2`	Master control node
`10.0.0.3+`	Replica control nodes
`10.0.0.x`	Worker nodes (offset after replicas)
`10.0.0.254`	Admin node (temporary)

IPv6-Only

Nodes have no public IPv4 addresses. The master control node always has public IPv6 (required for cloudflared). Replica control nodes and workers can optionally have public IPv6 disabled via enable_public_ipv6 = false to air-gap them from the internet. NAT64/DNS64 provides transparent IPv4 reachability for accessing IPv4-only services (GitHub, container registries, package repos).

Traffic Flow

SSH access: Internet → Cloudflare Tunnel → Master control node (localhost:22) → Private network → Other nodes
Outbound (control nodes): Full outbound connectivity via IPv6 + NAT64
Outbound (workers): Restricted to DNS, HTTP/S only

3.2 - Security Model

The three security layers: Hetzner firewalls, host hardening, and Kubernetes network policies.

KubeClaw’s security is layered across three levels: Hetzner Cloud firewalls, host-level hardening, and Kubernetes network policies. See Architecture for the overall network design.

Firewall Rules (Hetzner)

Hetzner Cloud firewalls are the first line of defense. Each node role has its own firewall:

Control nodes: SSH from private network + localhost (for tunnel), ICMP from private network
Worker nodes: SSH from private network, ICMP from private network
Admin node: SSH from anywhere (temporary), ICMP from private network

Kubernetes ports (6443, 10250, 2379-2380, 30000-32767) are intentionally excluded from the Hetzner firewall and added only when deploying Kubernetes.

Host-Level Security

SSH hardening: key-only auth, no root login, limited retries, no TCP forwarding on workers
fail2ban: SSH brute-force protection on all nodes
UFW: host-level firewall enforcing the same rules as the Hetzner firewall

For operational security details (SSH configuration directives, monitoring, security auditing), see Operations: Security.

Kubernetes-Level Security

Cilium CNI: eBPF-based network policies with FQDN egress filtering
Namespace isolation: system-unrestricted (full egress) and apps-restricted (whitelist-only egress)
Container hardening: non-root users, dropped capabilities, resource limits

3.3 - DNS and NAT64

How DNS64 and NAT64 give IPv6-only nodes and pods transparent access to IPv4-only services.

The cluster is IPv6-only – but many services (GitHub CDN, container registries, package repos) are IPv4-only. NAT64/DNS64 provides transparent IPv4 reachability at the network layer, no application changes needed. This page covers how DNS64/NAT64 works, how it integrates with Kubernetes CoreDNS, and how to configure it.

The Problem: IPv4 Internet from IPv6-Only Nodes

The nodes have no public IPv4 addresses. Most internet services (GitHub, Docker Hub, package repos) have IPv4 addresses. How does an IPv6-only node reach them?

┌──────────────────────────────────────────────────────────────────┐
│  The Problem                                                      │
│                                                                    │
│  Node (IPv6 only)  ──────╳──────►  github.com (140.82.121.3)    │
│  2a01:4f8:...              │        IPv4 address                  │
│                     No IPv4 route!                                 │
└──────────────────────────────────────────────────────────────────┘

The answer is DNS64 + NAT64, a standard mechanism (RFC 6146/6147) that gives IPv6-only clients transparent access to IPv4 servers.

How DNS64 + NAT64 Works (Node Level)

When a node needs to reach an IPv4-only service, the DNS64 resolver synthesizes a special IPv6 address that embeds the IPv4 address:

┌──────────────────────────────────────────────────────────────────┐
│  Step 1: DNS64 Resolution                                         │
│                                                                    │
│  Node                    DNS64 Resolver              Authoritative │
│  ┌──────┐               ┌──────────────┐             DNS Server   │
│  │      │──"github.com"─►│              │──"github.com"──►┌─────┐ │
│  │      │  "AAAA?"       │  2001:67c:   │  "AAAA?"        │     │ │
│  │      │                │  2b0::4      │◄── no AAAA ─────│     │ │
│  │      │                │              │──"github.com"──►│     │ │
│  │      │                │              │  "A?"            │     │ │
│  │      │                │              │◄── 140.82.121.3 ─│     │ │
│  │      │                │              │                  └─────┘ │
│  │      │                │  Synthesize: │                          │
│  │      │◄───────────────│  64:ff9b::   │                          │
│  │      │  AAAA record:  │  8c52:7903   │                          │
│  └──────┘  64:ff9b::     └──────────────┘                          │
│            8c52:7903                                               │
│                                                                    │
│  The DNS64 resolver embeds the IPv4 address (140.82.121.3 =       │
│  0x8c527903) into the well-known NAT64 prefix 64:ff9b::/96.      │
└──────────────────────────────────────────────────────────────────┘

┌──────────────────────────────────────────────────────────────────┐
│  Step 2: NAT64 Translation                                        │
│                                                                    │
│  Node                   NAT64 Gateway              github.com     │
│  ┌──────┐              (nat64.net)                 ┌──────────┐  │
│  │      │──IPv6 pkt───►┌──────────────┐──IPv4 pkt─►│          │  │
│  │      │  dst:        │  Recognizes   │  dst:      │140.82.   │  │
│  │      │  64:ff9b::   │  64:ff9b::    │  140.82.   │121.3     │  │
│  │      │  8c52:7903   │  prefix,      │  121.3     │          │  │
│  │      │              │  extracts     │            │          │  │
│  │      │◄─IPv6 pkt───│  IPv4 addr,   │◄─IPv4 pkt──│          │  │
│  └──────┘              │  translates   │            └──────────┘  │
│                        └──────────────┘                           │
│                                                                    │
│  The node sends IPv6 traffic to 64:ff9b::8c52:7903. The NAT64   │
│  gateway strips the prefix, extracts 140.82.121.3, and forwards  │
│  the packet as regular IPv4. Responses are translated back.      │
└──────────────────────────────────────────────────────────────────┘

Key detail: DNS64 only synthesizes AAAA records for domains that have no native AAAA record. If a domain already has an IPv6 address (like google.com), the DNS64 resolver returns the real AAAA record and no synthesis happens.

DNS Inside the Kubernetes Cluster (Dual-Stack)

With dual-stack networking, every pod gets both an IPv4 address (for internal cluster communication) and an IPv6 address (for external access via DNS64/NAT64). This eliminates the need for hostNetwork workarounds:

┌──────────────────────────────────────────────────────────────────┐
│  DNS Consumers and Their Resolvers                                │
│                                                                    │
│  ┌─────────────────────────────────────────────────────────────┐  │
│  │  Host processes (cloudflared, apt, curl)                     │  │
│  │  /etc/resolv.conf:                                           │  │
│  │    nameserver 10.96.0.10 ◄── CoreDNS ClusterIP              │  │
│  │    nameserver 2a01:4ff:ff00::add:2 ◄── Hetzner DNS          │  │
│  └─────────────────────────────────────────────────────────────┘  │
│                                                                    │
│  ┌─────────────────────────────────────────────────────────────┐  │
│  │  All pods (CoreDNS, CSI, OpenClaw, nginx, ...)              │  │
│  │  /etc/resolv.conf:                                           │  │
│  │    nameserver 10.96.0.10  ◄── CoreDNS ClusterIP             │  │
│  │    search apps-restricted.svc.cluster.local                  │  │
│  │           svc.cluster.local cluster.local                    │  │
│  │                                                               │  │
│  │  Pod IPs: 10.244.x.x (IPv4) + fd00:10:244::x (IPv6)        │  │
│  │  External access: via IPv6 → DNS64/NAT64 → IPv4 internet    │  │
│  └─────────────────────────────────────────────────────────────┘  │
└──────────────────────────────────────────────────────────────────┘

CoreDNS: The Bridge Between Cluster and External DNS

CoreDNS runs as a regular pod with dual-stack addresses. It forwards external queries to DNS64 resolvers, which synthesize AAAA records for IPv4-only domains. This allows all pods to reach external services via NAT64:

┌──────────────────────────────────────────────────────────────────┐
│  CoreDNS Query Flow                                               │
│                                                                    │
│                        CoreDNS (10.96.0.10)                       │
│                        dual-stack pod                              │
│                        ┌────────────────────┐                     │
│                        │                    │                     │
│  Pod queries           │  1. kubernetes     │                     │
│  "nginx.apps-          │     plugin         │                     │
│   restricted.svc.  ───►│     ↓              │                     │
│   cluster.local"       │  Matches           │  ──► 10.111.194.145│
│                        │  cluster.local?    │      (ClusterIP)    │
│                        │  YES → respond     │                     │
│                        │  from K8s API      │                     │
│                        │                    │                     │
│  Pod queries           │  2. forward        │                     │
│  "github.com"      ───►│     plugin         │                     │
│                        │     ↓              │                     │
│                        │  Matches           │                     │
│                        │  cluster.local?    │                     │
│                        │  NO → forward to   │                     │
│                        │  DNS64 resolvers   │──► DNS64 resolver   │
│                        │  2001:67c:2b0::4   │    2001:67c:2b0::4 │
│                        └────────────────────┘    ↓                │
│                                                  Synthesizes AAAA │
│                                                  64:ff9b::8c52:..│
└──────────────────────────────────────────────────────────────────┘

How Pods Reach External Services (Dual-Stack + NAT64)

With dual-stack, pods have IPv6 addresses and can route to the NAT64 prefix. Cilium’s IPv6 masquerading translates pod source addresses to the node’s public IPv6:

┌──────────────────────────────────────────────────────────────────┐
│  Pod Network Connectivity (Dual-Stack)                            │
│                                                                    │
│  Pod (10.244.0.5 + fd00:10:244::5)                               │
│  ├── ✅ → 10.96.0.10 (CoreDNS ClusterIP) ── DNS works            │
│  ├── ✅ → 10.111.194.145 (nginx ClusterIP) ── service routing    │
│  ├── ✅ → 10.0.0.2 (node private IP) ── host reachable           │
│  ├── ✅ → 64:ff9b::8c52:7903 (github.com via NAT64) ── works!   │
│  └── ✅ → 2a00:1450:... (google.com native IPv6) ── works!       │
│                                                                    │
│  Flow for IPv4-only destinations (e.g., api.anthropic.com):      │
│  1. Pod queries CoreDNS → forwards to DNS64 resolver              │
│  2. DNS64 synthesizes: api.anthropic.com → 64:ff9b::6812:0000   │
│  3. Cilium DNS proxy records the FQDN→IP mapping                 │
│  4. Pod sends IPv6 to 64:ff9b::6812:0000                        │
│  5. Cilium masquerades src to node's public IPv6                  │
│  6. NAT64 gateway translates to IPv4 → reaches api.anthropic.com │
│                                                                    │
│  Cilium FQDN policies enforce egress at every step:              │
│  toFQDNs: "api.anthropic.com" → allows 64:ff9b::6812:0000       │
│  All other external traffic is DENIED.                            │
└──────────────────────────────────────────────────────────────────┘

Full DNS Resolution Example: Pod Resolves an External Name

Here’s the complete flow when a pod in apps-restricted queries api.anthropic.com:

┌──────────────────────────────────────────────────────────────────┐
│  Complete DNS Flow: Pod → CoreDNS → DNS64 → NAT64                │
│                                                                    │
│  1. Pod sends DNS query                                           │
│     src: 10.244.0.5 → dst: 10.96.0.10:53                        │
│     "What is api.anthropic.com?"                                  │
│          │                                                        │
│          ▼                                                        │
│  2. Cilium DNS proxy intercepts the query                        │
│     Records the FQDN for policy matching                         │
│     Forwards to CoreDNS                                           │
│          │                                                        │
│          ▼                                                        │
│  3. CoreDNS receives query                                        │
│     kubernetes plugin: "api.anthropic.com" ≠ cluster.local        │
│     forward plugin: forward to DNS64 resolver (2001:67c:2b0::4)  │
│          │                                                        │
│          ▼                                                        │
│  4. DNS64 resolver synthesizes AAAA record                        │
│     api.anthropic.com has only A records (104.18.x.x)            │
│     Synthesizes: AAAA 64:ff9b::6812:0000                         │
│          │                                                        │
│          ▼                                                        │
│  5. Cilium DNS proxy records the mapping                          │
│     api.anthropic.com → 64:ff9b::6812:0000                       │
│     toFQDNs rule "api.anthropic.com" now allows this IP          │
│          │                                                        │
│          ▼                                                        │
│  6. Pod connects to 64:ff9b::6812:0000:443                       │
│     Cilium checks egress policy → ALLOWED (FQDN match)           │
│     IPv6 masquerade: src becomes node's public IPv6              │
│     NAT64 gateway translates to IPv4 104.18.x.x                  │
│     ✅ SUCCESS -- pod reaches api.anthropic.com                   │
└──────────────────────────────────────────────────────────────────┘

Full DNS Resolution Example: Host Resolves a Cluster Service

Here’s the flow when cloudflared (system service on the host) needs to reach a Kubernetes service:

┌──────────────────────────────────────────────────────────────────┐
│  Complete DNS Flow: cloudflared → CoreDNS → Kubernetes API        │
│                                                                    │
│  1. cloudflared queries the host's DNS                            │
│     /etc/resolv.conf: nameserver 10.96.0.10 (CoreDNS ClusterIP)  │
│     "What is nginx.apps-restricted.svc.cluster.local?"            │
│          │                                                        │
│          ▼                                                        │
│  2. CoreDNS receives query                                        │
│     kubernetes plugin: matches *.svc.cluster.local                │
│     Queries Kubernetes API for Service "nginx" in                 │
│     namespace "apps-restricted"                                   │
│          │                                                        │
│          ▼                                                        │
│  3. CoreDNS returns ClusterIP                                     │
│     nginx.apps-restricted.svc.cluster.local → 10.111.194.145     │
│          │                                                        │
│          ▼                                                        │
│  4. cloudflared connects to 10.111.194.145:80                    │
│     Cilium routes the ClusterIP to the nginx pod                  │
│     ✅ SUCCESS -- host can reach ClusterIP via Cilium             │
└──────────────────────────────────────────────────────────────────┘

Summary: Who Resolves What

Consumer	Resolver	Cluster names	External names	External connectivity
Host processes (cloudflared, apt)	CoreDNS (ClusterIP) + Hetzner DNS	Yes (via CoreDNS)	Yes (via Hetzner DNS)	Full (IPv6 + NAT64)
All pods (CoreDNS, CSI, OpenClaw, etc.)	CoreDNS (10.96.0.10 ClusterIP)	Yes	Yes (AAAA synthesized via DNS64)	Full (IPv6 + NAT64 via masquerade)
Node itself (DNS64 configured)	DNS64 resolvers (2001:67c:2b0::4)	No	Yes (AAAA synthesized)	Full (IPv6 + NAT64)

Configuration

NAT64/DNS64 is enabled by default (enable_nat64 = true). Cloud-init configures it on new nodes automatically. Default resolvers are from nat64.net (Nuremberg, Helsinki, Amsterdam) – close to Hetzner’s fsn1 datacenter.

For existing nodes

Run the Ansible playbook:

cd ansible
ansible-playbook playbooks/configure-nat64.yml

Limit to specific node groups:

ansible-playbook playbooks/configure-nat64.yml --limit control_nodes

Override resolvers:

ansible-playbook playbooks/configure-nat64.yml \
  -e '{"dns64_resolvers":["2a01:4f8:c2c:123f::1"]}'

Disabling NAT64

If you set up your own DNS infrastructure:

# terraform.tfvars
enable_nat64 = false

Verification

# DNS64 synthesis (should show AAAA record with 64:ff9b:: prefix)
resolvectl query github.com

# End-to-end connectivity
curl -6 https://github.com

Technical Details

What cloud-init configures

DNS64 resolvers in /etc/systemd/resolved.conf.d/dns64.conf
NAT64 route: 64:ff9b::/96 via the default IPv6 gateway
networkd-dispatcher script to persist the route across reboots

What the Ansible playbook configures

The same as cloud-init, plus:

Removes old Hetzner DNS UFW rules on worker nodes
Adds DNS64 resolver allow rules in UFW (workers only)
Adds NAT64 prefix UFW rule (workers only)
Verifies DNS64 resolution and NAT64 connectivity

Worker node specifics

Worker nodes have restricted outbound access. The NAT64 configuration adds:

UFW rules allowing DNS to DNS64 resolvers (instead of Hetzner DNS)
UFW rule allowing traffic to the 64:ff9b::/96 prefix

4 - Guide

Deploy and operate the KubeClaw infrastructure step by step.

Follow these guides to provision the infrastructure, configure the nodes, and deploy OpenClaw on Kubernetes.

4.1 - Dev Container

Set up the aibox Dev Container that carries every tool this project needs.

The project uses an aibox-generated Dev Container (Debian Trixie) as a self-contained, reproducible environment. Nothing needs to be set up on the host machine beyond Docker and an IDE with Dev Container support.

What the Dev Container Provides

OpenTofu – infrastructure provisioning (Terraform-compatible)
Ansible – server management (runs inside the container, no host install needed)
cloudflared – SSH ProxyCommand via Cloudflare Tunnel
ssh-agent – start inside the container to use passphrase-protected keys with Ansible
AI CLI tools – Codex and the aibox toolchain
Hugo + Docsy – documentation site (./scripts/serve-docs.sh at port 1313)
Persistence – .aibox-home/.ssh/ is bind-mounted read-write by this project, so setup-ssh.sh output, SSH config, and exported keys survive container rebuilds

Step 1: Prepare Persistent Directories

Create the persistent directories before opening the Dev Container:

git clone <repo-url>
cd kubeclaw

# Create the persistent SSH directory mounted at /home/aibox/.ssh.
mkdir -p .aibox-home/.ssh
chmod 700 .aibox-home/.ssh

The .aibox-home/ directory is gitignored. Its .ssh/ subdirectory holds your private keys, SSH configuration, and known_hosts without risking a commit of secrets.

Step 2: Open in Dev Container

1. Install the [Dev Containers](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers) extension
2. Open the project folder in VS Code
3. Click **"Reopen in Container"** when prompted, or press ++cmd+shift+p++ and select **Dev Containers: Reopen in Container**

The generated setup uses Docker Compose. Use your editor’s Dev Container command rather than docker build directly so the generated Compose mounts, including the persistent aibox home directory, are applied.

Step 3: Start SSH Agent

source ./scripts/ssh-agent-setup.sh

Note

No SSH keys exist yet at this point -- they are created later during [Infrastructure provisioning](/kubeclaw/docs/guide/infrastructure/). This step starts the ssh-agent and fixes SSH directory permissions, preparing the environment for later steps.

Important

The `source` command is required -- running the script without `source` starts the agent in a subprocess that terminates immediately.

SSH persistence

Docker bind mounts from macOS may not preserve Unix permissions. The `ssh-agent-setup.sh` script automatically fixes SSH directory permissions on every run. The `.aibox-home/.ssh/` directory stores SSH private keys, SSH config, and `known_hosts` -- all exported or generated by `setup-ssh.sh`. Everything survives container rebuilds.

Step 4: Verify Setup

Confirm that the essential tools are available:

tofu --version
ansible --version
hugo version
npm --version

Preview the documentation:

./scripts/serve-docs.sh
# Open http://localhost:1313 in your browser

4.2 - Infrastructure (OpenTofu)

Configure terraform.tfvars, run tofu apply, and set up SSH access to the new nodes.

KubeClaw uses OpenTofu (Terraform-compatible) to provision all infrastructure on Hetzner Cloud. This page walks through the provisioning workflow step by step. See Variables Reference for all configurable options.

Step 1: Configure terraform.tfvars

cp terraform.tfvars.example terraform.tfvars

Edit terraform.tfvars with your settings:

hcloud_token    = "your-hcloud-api-token"
root_password   = "a-strong-root-password"
cluster_name    = "k8s-cluster"

# Optional: auto-configure Cloudflare Tunnel on the master control node
# cloudflare_tunnel_token = "eyJ..."

SSH key options

No configuration needed. OpenTofu generates ED25519 keys and stores them in the state file.

| Advantage | Disadvantage |
|-----------|--------------|
| No manual key creation | State file contains private keys |
| Works out of the box | Keys lost if state is lost |

Create your own keys and reference them in `terraform.tfvars`:

```bash
ssh-keygen -t ed25519 -f ~/.ssh/k8s-cluster_control-node_key -C "control-node"
ssh-keygen -t ed25519 -f ~/.ssh/k8s-cluster_worker-node_key -C "worker-node"
```

```hcl
control_node_public_key = "ssh-ed25519 AAAA... control-node"
worker_node_public_key  = "ssh-ed25519 AAAA... worker-node"
```

| Advantage | Disadvantage |
|-----------|--------------|
| Full control over key storage | Manual key management |
| State has no private keys | Must create keys before provisioning |
| Easy password manager integration | |

Step 2: Provision Infrastructure

tofu init    # First time only
tofu apply

This creates:

Private network (10.0.0.0/24)
Role-specific firewalls (control node, worker node, admin node)
SSH keys (uploaded to Hetzner)
Admin node at 10.0.0.254 (if enable_admin_node = true)
Master control node at 10.0.0.2 (with cloudflared if tunnel token is set)
Replica control nodes and workers (if configured)

Step 3: Set Up SSH Access

./scripts/setup-ssh.sh

This exports SSH private keys from OpenTofu state and generates ~/.ssh/config entries. Test the connection:

ssh control-node

Tip

If you used `cloudflare_tunnel_token`, SSH routes through Cloudflare Tunnel automatically. Otherwise, the admin node serves as a jump host during initial setup.

Next Steps

Server Management (Ansible) – generate the Ansible inventory, apply system updates and security hardening
Kubernetes (kubeadm) – deploy the cluster

SSH Keys with Passphrase – add passphrase protection to SSH keys
Store SSH Keys in Password Manager – backup keys securely
SSH Key Rotation – rotate keys periodically or after compromise
Scale Up/Down – add or remove nodes
OS Images – available Hetzner Cloud images

4.3 - Cloudflare Tunnel Setup

Create the tunnel, route SSH through it, and protect it with a Cloudflare Access policy.

This guide walks through creating and configuring a Cloudflare Tunnel that provides secure SSH access to your KubeClaw cluster. The tunnel replaces the temporary admin node as the permanent access path – no open ports, no public SSH, outbound-only connectivity.

Why Cloudflare Tunnel?

KubeClaw nodes have no public IPv4 and no inbound SSH ports. Access works through one of two paths:

Path	When to use	How it works
Admin node (temporary)	Initial setup, before tunnel is ready	Jump host with public IPv6; SSH via `ProxyJump`
Cloudflare Tunnel (permanent)	After tunnel is configured	`cloudflared` on the master node connects outbound to Cloudflare’s edge; SSH proxied via `ProxyCommand cloudflared access ssh` on your local machine

After the tunnel is working, you disable the admin node (enable_admin_node = false in terraform.tfvars) and all SSH flows through Cloudflare.

cloudflared on both sides

`cloudflared` runs in two places, because there is no direct SSH path to the cluster. The hostname `console.yourdomain.org` points to Cloudflare's edge servers, which only speak HTTPS -- a plain `ssh console.yourdomain.org` on port 22 would be refused. Instead, the SSH byte stream is wrapped inside HTTPS and carried through the tunnel: 1. **Your machine / Dev Container** (client side): The SSH config uses `ProxyCommand cloudflared access ssh --hostname %h`. Instead of opening a TCP connection, SSH pipes its traffic through this subprocess. `cloudflared` connects to Cloudflare's edge over HTTPS (port 443), handles Cloudflare Access authentication (browser-based, if configured), and forwards the SSH bytes. 2. **Cloudflare edge**: Validates the Access JWT, looks up which tunnel serves the hostname, and forwards the traffic to the matching tunnel connector. 3. **Master control node** (server side): The server-side `cloudflared` maintains a persistent outbound connection to Cloudflare's edge. It receives the forwarded traffic and proxies it to `localhost:22`, where sshd handles normal key-based authentication. The full chain: `ssh` ↔ `cloudflared (local)` ↔ `Cloudflare edge (HTTPS)` ↔ `cloudflared (server)` ↔ `sshd`. The Dev Container includes cloudflared. On macOS, install it with `brew install cloudflared`.

Prerequisites

A Cloudflare account (free tier is sufficient)
A domain added to Cloudflare (Cloudflare must be the DNS provider)
Infrastructure provisioned with tofu apply (the master control node must be running)

Step 1: Open Zero Trust Dashboard

Go to https://one.dash.cloudflare.com and log in. This opens the Cloudflare Zero Trust dashboard (formerly Cloudflare for Teams).

┌──────────────────────────────────────────────────────────────────┐
│  Cloudflare | Zero Trust                                         │
├──────────────┬───────────────────────────────────────────────────┤
│              │                                                   │
│  Home        │   Zero Trust Overview                             │
│  Analytics   │                                                   │
│  Risk Score  │   ┌──────────┐  ┌──────────┐  ┌──────────┐      │
│              │   │ Users    │  │ Tunnels  │  │ Policies │      │
│  Access ►    │   │ 0        │  │ 0        │  │ 0        │      │
│  Gateway     │   └──────────┘  └──────────┘  └──────────┘      │
│  Networks ►  │                                                   │
│  ...         │                                                   │
│              │                                                   │
└──────────────┴───────────────────────────────────────────────────┘

Step 2: Create a Tunnel

In the left sidebar, navigate to Networks > Tunnels
Click Create a tunnel

┌──────────────────────────────────────────────────────────────────┐
│  Networks > Tunnels                                              │
│                                                                  │
│  ┌────────────────────────────────────────────────────────────┐  │
│  │                   Create a tunnel                          │  │
│  │                                                            │  │
│  │  Select your tunnel type:                                  │  │
│  │                                                            │  │
│  │  ┌─────────────────────┐   ┌─────────────────────┐       │  │
│  │  │ ● Cloudflared       │   │ ○ WARP Connector     │       │  │
│  │  │   (recommended)     │   │                      │       │  │
│  │  └─────────────────────┘   └─────────────────────┘       │  │
│  │                                                  [ Next ] │  │
│  └────────────────────────────────────────────────────────────┘  │
└──────────────────────────────────────────────────────────────────┘

Select Cloudflared and click Next
Enter a tunnel name, e.g. kubeclaw or hetzner-cluster

┌──────────────────────────────────────────────────────────────────┐
│  Name your tunnel                                                │
│                                                                  │
│  Tunnel name:  ┌──────────────────────────────┐                 │
│                │ kubeclaw                      │                 │
│                └──────────────────────────────┘                 │
│                                                                  │
│                                               [ Save tunnel ]   │
└──────────────────────────────────────────────────────────────────┘

Click Save tunnel

Step 3: Copy the Tunnel Token

After saving, Cloudflare shows the connector installation instructions. The page displays install commands for various platforms. You need the token value from the install command.

┌──────────────────────────────────────────────────────────────────┐
│  Install and run a connector                                     │
│                                                                  │
│  Choose your environment:                                        │
│  [ Debian ] [ Docker ] [ macOS ] [ Windows ]                    │
│                                                                  │
│  Install and run a connector:                                    │
│  ┌────────────────────────────────────────────────────────────┐  │
│  │ sudo cloudflared service install eyJhIjoiY2Y...long-token │  │
│  └────────────────────────────────────────────────────────────┘  │
│                                                           📋     │
│                                                                  │
│  The token is: eyJhIjoiY2Y...                                   │
│                                                                  │
│                                                    [ Next ]      │
└──────────────────────────────────────────────────────────────────┘

Copy the token (the eyJ... string). You will need it in two places:

terraform.tfvars — so cloud-init auto-installs the tunnel on the master node
Manual install — if the infrastructure is already running

Save the token securely

The tunnel token is a long-lived credential. Store it in your password manager alongside the Hetzner API token.

Click Next to proceed to the hostname configuration.

Step 4: Add a Public Hostname for SSH

This step maps a subdomain to the SSH service on your master control node.

You should now be on the Route tunnel screen, or navigate to your tunnel’s Public Hostname tab
Click Add a public hostname
Fill in the hostname configuration:

┌──────────────────────────────────────────────────────────────────┐
│  Public Hostnames > Add a public hostname                        │
│                                                                  │
│  Public hostname                                                 │
│  ┌──────────────┐   ┌─────────────────────────┐                 │
│  │ console      │ . │ yourdomain.org       ▼  │                 │
│  │ (subdomain)  │   │ (domain)                │                 │
│  └──────────────┘   └─────────────────────────┘                 │
│                                                                  │
│  Path (optional):  ┌──────────────────────────┐                 │
│                    │                          │                 │
│                    └──────────────────────────┘                 │
│                                                                  │
│  Service                                                         │
│  ┌──────────────┐   ┌─────────────────────────┐                 │
│  │ SSH       ▼  │   │ localhost:22            │                 │
│  │ (type)       │   │ (URL)                   │                 │
│  └──────────────┘   └─────────────────────────┘                 │
│                                                                  │
│                                              [ Save hostname ]  │
└──────────────────────────────────────────────────────────────────┘

Field	Value	Notes
Subdomain	`console`	Or any name you prefer (e.g. `ssh`, `cluster`)
Domain	Your Cloudflare-managed domain	Must be a domain with Cloudflare DNS
Path	(leave empty)	Not used for SSH
Type	`SSH`	From the dropdown
URL	`localhost:22`	The tunnel connector runs on the master node, so SSH is on localhost

Click Save hostname

The resulting hostname (e.g. console.yourdomain.org) is what you’ll use as cloudflare_tunnel_domain in terraform.tfvars and as the SSH Host in your SSH config.

Step 5: Create an Access Application (Optional but Recommended)

Cloudflare Access adds browser-based authentication before the SSH connection is established. Without it, anyone who knows your tunnel hostname can attempt SSH connections (still protected by SSH keys, but Access adds a second layer).

In the left sidebar, go to Access > Applications
Click Add an application
Select Self-hosted

┌──────────────────────────────────────────────────────────────────┐
│  Access > Applications > Add an application                      │
│                                                                  │
│  Application Configuration                                       │
│                                                                  │
│  Application name:  ┌─────────────────────────┐                 │
│                     │ KubeClaw SSH Console     │                 │
│                     └─────────────────────────┘                 │
│                                                                  │
│  Session Duration:  ┌─────────────────────────┐                 │
│                     │ 24 hours             ▼  │                 │
│                     └─────────────────────────┘                 │
│                                                                  │
│  Application domain:                                             │
│  ┌──────────────┐   ┌─────────────────────────┐                 │
│  │ console      │ . │ yourdomain.org       ▼  │                 │
│  └──────────────┘   └─────────────────────────┘                 │
│                                                                  │
│                                                    [ Next ]      │
└──────────────────────────────────────────────────────────────────┘

Click Next to configure a policy
Create an Allow policy:

┌──────────────────────────────────────────────────────────────────┐
│  Add Policies                                                    │
│                                                                  │
│  Policy name:  ┌──────────────────────────┐                     │
│                │ Allow Admin              │                     │
│                └──────────────────────────┘                     │
│                                                                  │
│  Action:  ┌──────────────────────────┐                          │
│           │ Allow                 ▼  │                          │
│           └──────────────────────────┘                          │
│                                                                  │
│  Configure rules:                                                │
│  ┌────────────────────────────────────────────────────────────┐  │
│  │  Include                                                   │  │
│  │  Selector:  ┌─────────────────┐  Value: ┌───────────────┐ │  │
│  │             │ Emails       ▼  │         │ you@email.com │ │  │
│  │             └─────────────────┘         └───────────────┘ │  │
│  └────────────────────────────────────────────────────────────┘  │
│                                                                  │
│                                                    [ Next ]      │
└──────────────────────────────────────────────────────────────────┘

Field	Value
Policy name	`Allow Admin`
Action	`Allow`
Include selector	`Emails`
Include value	Your email address

Click Next, review, and Save

Authentication flow

On first SSH connection, `cloudflared access ssh` opens a browser window. You authenticate via Cloudflare Access (email OTP, or your configured identity provider). After authentication, SSH proceeds normally with key-based auth. The browser session lasts for the configured session duration.

After completing the Cloudflare setup, continue with Infrastructure (OpenTofu) to configure the tunnel token and domain in terraform.tfvars, provision the cluster, and verify connectivity.

How SSH Routing Works

After the tunnel is active and the admin node is disabled:

┌──────────────┐     ┌──────────────┐     ┌──────────────────────┐
│  Your        │     │  Cloudflare  │     │  Master Control Node │
│  machine     │────►│  Edge        │────►│  (cloudflared →      │
│  (cloudflared│     │  Network     │     │   localhost:22)      │
│   access ssh)│     │              │     │                      │
└──────────────┘     └──────────────┘     └──────────┬───────────┘
                                                     │ ProxyJump
                                          ┌──────────▼───────────┐
                                          │  Replicas / Workers  │
                                          │  (10.0.0.3+)         │
                                          └──────────────────────┘

Target	SSH command	Route
Master control node	`ssh console.yourdomain.org`	ProxyCommand → Cloudflare → cloudflared → localhost:22
Replica control node	`ssh control-02`	ProxyCommand → Cloudflare → master → ProxyJump → 10.0.0.3
Worker node	`ssh worker-01`	ProxyCommand → Cloudflare → master → ProxyJump → 10.0.0.x

Troubleshooting

Tunnel shows “Inactive” or “Down”

# On the master control node:
sudo systemctl status cloudflared
sudo journalctl -u cloudflared --no-pager -n 50

Common causes:

Token expired or revoked – create a new tunnel and update terraform.tfvars
DNS64/NAT64 not working – cloudflared needs outbound connectivity; check with curl -6 https://cloudflare.com

Browser authentication loop

If ssh console.yourdomain.org keeps opening the browser without connecting:

# Clear cached credentials
cloudflared access login --reset console.yourdomain.org

“connection refused” after authentication

The SSH service on the master node may not be listening on localhost:

# On the master node, verify SSH listens on 127.0.0.1
sudo ss -tlnp | grep 22

The cloud-init template configures UFW to allow SSH from localhost for exactly this reason.

Next Steps

Infrastructure (OpenTofu) – provision the cluster (if not done yet)
Server Management (Ansible) – apply updates and hardening
Kubernetes (kubeadm) – deploy the cluster

4.4 - Server Management (Ansible)

Generate the inventory and run the update, hardening, NAT64, and Kubernetes-prerequisite playbooks.

Ansible handles ongoing server management: system updates, security hardening, NAT64 configuration, and Kubernetes prerequisites. All playbooks run from the Dev Container.

Step 1: Set Up Ansible

1.1 Generate inventory

./scripts/generate-ansible-inventory.sh

The inventory (ansible/inventory.ini) is auto-generated. It defines three groups:

control_nodes – all control plane nodes
worker_nodes – all worker nodes
k8s_cluster – union of control and worker nodes

Warning

Never hand-edit `inventory.ini`. Regenerate after infrastructure changes with `./scripts/generate-ansible-inventory.sh`.

1.2 Load SSH keys

source ./scripts/ssh-agent-setup.sh

Important

The ssh-agent must be running with the cluster keys loaded before Ansible can connect. Run `source ./scripts/ssh-agent-setup.sh` in every new terminal session.

1.3 Test connectivity

cd ansible
ansible all -m ping

Step 2: Apply System Updates

ansible-playbook playbooks/update-system.yml

Target specific node groups or enable automatic reboots:

# Only control nodes
ansible-playbook playbooks/update-system.yml --limit control_nodes

# With reboot if kernel was updated
ansible-playbook playbooks/update-system.yml -e "reboot_after_update=true"

Step 3: Apply Security Hardening

ansible-playbook playbooks/security-hardening.yml

This enables:

Unattended upgrades (automatic security updates)
fail2ban monitoring
Kernel security parameters (sysctl hardening)
Secure permissions on sensitive files
Core dump disabling

Step 4: Configure NAT64/DNS64

For existing nodes that weren’t configured via cloud-init:

ansible-playbook playbooks/configure-nat64.yml

See DNS and NAT64 for details.

Step 5: Install Kubernetes Prerequisites

ansible-playbook playbooks/prepare-k8s-nodes.yml

This installs the selected container runtime, kubeadm, kubelet, and kubectl on all nodes. The runtime defaults to containerd; pass -e "container_runtime=cri-o" to use CRI-O instead, matching the container_runtime variable used by cloud-init.

Next Steps

Kubernetes (kubeadm) – deploy the cluster
Ansible Playbooks Reference – detailed playbook documentation

4.5 - Kubernetes (kubeadm)

Bootstrap a dual-stack kubeadm cluster with Cilium CNI and the Hetzner CSI driver.

After provisioning the infrastructure with OpenTofu and configuring SSH access, deploy a standard Kubernetes cluster using kubeadm with Cilium as the CNI.

Why kubeadm + Cilium?

kubeadm: The official Kubernetes bootstrapper. Produces a standard, upstream cluster – exactly what the CKA exam expects. Full control over every component (etcd, kube-apiserver, kube-scheduler, kube-controller-manager).
Cilium: eBPF-based CNI providing advanced network policies with FQDN-based egress filtering – critical for restricting outbound traffic per namespace.

Step 1: Initialize the Control Plane (Dual-Stack)

SSH into the master control node. Prerequisites (containerd, kubeadm, kubelet, kubectl) are already installed via cloud-init when enable_k8s_prereqs = true (default).

First, determine the master’s private IP (default: 10.0.0.2, depends on subnet_ip_range):

# From the Dev Container:
tofu output -raw master_control_node_private_ip

The commands below use $MASTER_IP. Set it on the master node before proceeding:

MASTER_IP=10.0.0.2

Do not use `hostname -I`

`hostname -I` may return Hetzner's CGNAT address (`100.64.x.x`) as the first IP instead of the private network IP. Always set `MASTER_IP` explicitly to the private network address from your `subnet_ip_range` (default: `10.0.0.2`).

1.1 Verify prerequisites

IPv6 forwarding

sudo sysctl net.ipv6.conf.all.forwarding

Expected output: net.ipv6.conf.all.forwarding = 1

If not set, enable it:

echo "net.ipv6.conf.all.forwarding = 1" | sudo tee -a /etc/sysctl.d/k8s.conf
sudo sysctl --system

Kernel modules

lsmod | grep -E 'overlay|br_netfilter'

Expected output (both modules present):

br_netfilter           ...
overlay                ...

Sysctl parameters

sudo sysctl net.bridge.bridge-nf-call-iptables net.bridge.bridge-nf-call-ip6tables net.ipv4.ip_forward

Expected output:

net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1

Container runtime

```bash
systemctl is-active containerd
```

Expected output: `active`

```bash
systemctl is-active crio
```

Expected output: `active`

<div class="alert alert-primary" role="alert"><div class="h4 alert-heading" role="heading">CRI-O CNI path</div>


CRI-O uses `/opt/cni/bin` for CNI binaries (the upstream default), unlike Debian's containerd which uses `/usr/lib/cni`. When installing Cilium with CRI-O, use `--set cni.binPath=/opt/cni/bin` (or omit the flag, since `/opt/cni/bin` is Cilium's default).

</div>

kubeadm

kubeadm version -o yaml

Expected output (version numbers will vary):

clientVersion:
  gitVersion: v1.32.x
  platform: linux/amd64
  ...

1.2 Determine the node’s IPv6 address

Each node needs both an IPv4 and IPv6 address for dual-stack. The IPv4 is the private network IP (10.0.0.2). For IPv6, use the node’s public IPv6 address:

NODE_IPV6=$(ip -6 addr show scope global | grep -oP '(?<=inet6\s)[\da-f:]+' | head -1)
echo $NODE_IPV6

This should print a public IPv6 address like 2a01:4f8:xxxx:xxxx::1. Save this – you’ll need it for kubelet configuration.

1.3 Initialize with kubeadm (dual-stack)

Verify MASTER_IP before proceeding

Run `echo $MASTER_IP` -- it **must** print `10.0.0.2` (or your custom subnet IP). If it's empty, go back and set it. Running `kubeadm init` with an empty or wrong `--apiserver-advertise-address` will bind the API server to the wrong interface and generate TLS certificates with incorrect SANs. Fixing this requires a full `kubeadm reset` and re-init.

sudo kubeadm init \
  --apiserver-advertise-address=$MASTER_IP \
  --pod-network-cidr=10.244.0.0/16,fd00:10:244::/48 \
  --service-cidr=10.96.0.0/12,fd00:10:96::/108 \
  --skip-phases=addon/kube-proxy

Flags explained:

--apiserver-advertise-address=$MASTER_IP – Bind the API server to the private network IP (single address, not dual-stack)
--pod-network-cidr=10.244.0.0/16,fd00:10:244::/48 – Dual-stack pod CIDRs: IPv4 for internal cluster communication + IPv6 for external connectivity via DNS64/NAT64
--service-cidr=10.96.0.0/12,fd00:10:96::/108 – Dual-stack service CIDRs: existing IPv4 services continue to work; new services can opt into dual-stack
--skip-phases=addon/kube-proxy – Cilium replaces kube-proxy with eBPF datapath

CKA note

The CKA exam typically uses kube-proxy. This cluster skips it because Cilium provides a more efficient replacement. On the exam, omit `--skip-phases=addon/kube-proxy`.

Why dual-stack?

The nodes are IPv6-only with NAT64/DNS64 for IPv4 reachability. By giving pods IPv6 addresses alongside IPv4, they can reach external services via DNS64/NAT64 natively -- no `hostNetwork` workarounds needed. This means Cilium's FQDN-based egress policies apply to **all** pods, including CoreDNS, the CSI controller, and application workloads like OpenClaw.

CIDRs cannot be changed after init

kubeadm does not support modifying pod or service CIDRs after initialization. If you need different ranges, you must `kubeadm reset` and re-init.

Expected output (abbreviated, IPs and hashes will differ):

[init] Using Kubernetes version: v1.32.x
[preflight] Running pre-flight checks
...
[addons] Applied essential addon: CoreDNS

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join <MASTER_IP>:6443 --token <TOKEN> \
        --discovery-token-ca-cert-hash sha256:<HASH>

Expected warnings

**"remote version is much newer"** -- This is normal when pinning `kubernetes_version` (e.g., `1.32`). kubeadm detects a newer stable release exists but correctly falls back to the pinned version. No action needed. **"sandbox image is inconsistent"** -- The containerd default config ships `pause:3.8` but kubeadm 1.32 expects `pause:3.10`. The cloud-init and Ansible prerequisites already fix this. If you see this warning, update the sandbox image manually: `sudo sed -i 's|registry.k8s.io/pause:3.8|registry.k8s.io/pause:3.10|' /etc/containerd/config.toml && sudo systemctl restart containerd`

What kubeadm init does behind the scenes:

Generates PKI certificates (CA, API server, kubelet, etc.) in /etc/kubernetes/pki/
Writes static pod manifests for etcd, kube-apiserver, kube-controller-manager, kube-scheduler in /etc/kubernetes/manifests/
Bootstraps etcd and starts the API server
Configures RBAC and creates bootstrap tokens
Generates admin.conf kubeconfig for cluster administration

1.4 Set up kubeconfig

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

1.5 Configure kubelet for dual-stack

Set the kubelet’s --node-ip to both the IPv4 private network address and the node’s IPv6 address. This ensures endpoints are registered with both addresses:

sudo sed -i "s/KUBELET_KUBEADM_ARGS=\"/KUBELET_KUBEADM_ARGS=\"--node-ip=$MASTER_IP,$NODE_IPV6 /" /var/lib/kubelet/kubeadm-flags.env
sudo systemctl restart kubelet

Verify:

kubectl get nodes -o wide

The INTERNAL-IP column should show the IPv4 address. Check that both addresses are registered:

kubectl get nodes -o jsonpath='{.items[0].status.addresses}' | python3 -m json.tool

You should see both InternalIP entries (IPv4 and IPv6).

1.6 Save join command

# Print the join command (token valid for 24h)
kubeadm token create --print-join-command

Save this output – you’ll need it for worker nodes.

1.7 Verify

Node status

kubectl get nodes

The node shows NotReady until the CNI (Cilium) is installed in Step 3:

NAME                      STATUS     ROLES           AGE   VERSION
<cluster>-control-01      NotReady   control-plane   XXm   v1.32.x

System pods

kubectl get pods -n kube-system

Core control plane pods should be Running. CoreDNS pods remain Pending until the CNI is installed:

NAME                                          READY   STATUS    RESTARTS   AGE
coredns-xxxxxxxxxx-xxxxx                      0/1     Pending   0          XXm
coredns-xxxxxxxxxx-xxxxx                      0/1     Pending   0          XXm
etcd-<cluster>-control-01                     1/1     Running   0          XXm
kube-apiserver-<cluster>-control-01           1/1     Running   0          XXm
kube-controller-manager-<cluster>-control-01  1/1     Running   0          XXm
kube-scheduler-<cluster>-control-01           1/1     Running   0          XXm

Step 2: Join Worker Nodes and Further Control Nodes

Worker Nodes

SSH into each worker node and run the join command from Step 1.6. Before joining, configure the kubelet for dual-stack on the worker:

# On the worker node, determine its IPs
WORKER_IPV4=10.0.0.X   # Replace with the worker's private IP
WORKER_IPV6=$(ip -6 addr show scope global | grep -oP '(?<=inet6\s)[\da-f:]+' | head -1)

Then join:

sudo kubeadm join <MASTER_IP>:6443 --token <TOKEN> \
  --discovery-token-ca-cert-hash sha256:<HASH>

After joining, set the worker’s dual-stack node IP:

sudo sed -i "s/KUBELET_KUBEADM_ARGS=\"/KUBELET_KUBEADM_ARGS=\"--node-ip=$WORKER_IPV4,$WORKER_IPV6 /" /var/lib/kubelet/kubeadm-flags.env
sudo systemctl restart kubelet

CKA explainer -- TLS bootstrap

The worker uses the bootstrap token to authenticate with the API server, then requests a kubelet client certificate. The API server validates the token, signs the certificate, and the kubelet starts using it for all subsequent communication. This is the TLS bootstrap process.

For HA control plane (replica control nodes):

sudo kubeadm join <MASTER_IP>:6443 --token <TOKEN> \
  --discovery-token-ca-cert-hash sha256:<HASH> \
  --control-plane --certificate-key <CERT_KEY>

Generate the certificate key on the master: sudo kubeadm init phase upload-certs --upload-certs

Then configure --node-ip the same way as for workers.

Step 3: Install Cilium CNI (Dual-Stack)

3.1 Install Helm

On the control node, install Helm via the official install script:

curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

NAT64 and GitHub

The install script downloads from GitHub via NAT64. This generally works but connections can be flaky. If the download times out, simply retry the command.

3.2 Install Cilium via Helm

Add the Cilium Helm repository and install. Make sure $MASTER_IP is still set (echo $MASTER_IP) – if not, re-export it (see Step 1):

helm repo add cilium https://helm.cilium.io/
helm repo update

helm install cilium cilium/cilium \
  --version 1.16.5 \
  --namespace kube-system \
  --set kubeProxyReplacement=true \
  --set k8sServiceHost=$MASTER_IP \
  --set k8sServicePort=6443 \
  --set ipam.mode=kubernetes \
  --set ipv4.enabled=true \
  --set ipv6.enabled=true \
  --set enableIPv6Masquerade=true \
  --set operator.replicas=1 \
  --set cni.binPath=/usr/lib/cni

Flags explained:

--version 1.16.5 – Pin the Cilium version for reproducibility
--set kubeProxyReplacement=true – Replace kube-proxy with Cilium’s eBPF datapath (matches --skip-phases=addon/kube-proxy from Step 1)
--set k8sServiceHost / k8sServicePort – Required when kube-proxy is skipped, so Cilium knows how to reach the API server
--set ipam.mode=kubernetes – Use the Kubernetes host-scope IPAM. Without this, Cilium defaults to its own cluster-pool allocator with CIDRs 10.0.0.0/8 and fd00::/104, ignoring kubeadm’s --pod-network-cidr. This causes pod IPs to overlap with the Hetzner private network (10.0.0.0/24)
--set ipv4.enabled=true – Enable IPv4 pod networking (cluster-internal communication)
--set ipv6.enabled=true – Enable IPv6 pod networking (external access via DNS64/NAT64)
--set enableIPv6Masquerade=true – Masquerade pod IPv6 traffic to the node’s public IPv6 when leaving the cluster. This is what allows pods to reach external services via NAT64
--set operator.replicas=1 – Cilium defaults to 2 operator replicas, but since the operator uses a host port, only one can run per node. Set to 1 for single-node clusters; increase when adding worker nodes
--set cni.binPath=/usr/lib/cni – Debian’s containerd package looks for CNI binaries in /usr/lib/cni instead of the default /opt/cni/bin/. Without this, kubelet reports cni plugin not initialized. If using CRI-O, omit this flag (CRI-O uses the default /opt/cni/bin)

Restart containerd after Cilium install

After Cilium deploys the CNI plugin, containerd may have cached the "not initialized" state. Restart it to pick up the new CNI: ```bash sudo systemctl restart containerd ``` Wait a few seconds, then verify the node becomes `Ready` with `kubectl get nodes`.

3.3 Wait and verify

# Wait for the Cilium DaemonSet to roll out
kubectl -n kube-system rollout status daemonset/cilium --timeout=120s

# Verify Cilium pods are running
kubectl get pods -n kube-system -l k8s-app=cilium

After containerd restarts and Cilium is ready, all nodes should show Ready:

kubectl get nodes

Single-node cluster: removing the control-plane taint

By default, kubeadm applies a `NoSchedule` taint to control-plane nodes. This prevents regular workloads from running on nodes dedicated to the Kubernetes control plane (API server, etcd, scheduler), reserving their resources for cluster management. On a multi-node cluster this is the correct behavior. On a single control-plane node without workers, this taint prevents **all** pod scheduling (including CoreDNS), so it must be removed: ```bash kubectl taint nodes --all node-role.kubernetes.io/control-plane:NoSchedule- ``` If you later add worker nodes and want to restore the taint to keep workloads off the control plane: ```bash kubectl taint nodes node-role.kubernetes.io/control-plane:NoSchedule ```

3.4 Verify dual-stack pod connectivity

Once CoreDNS is running, verify that pods have both IPv4 and IPv6 addresses:

kubectl get pods -n kube-system -l k8s-app=kube-dns -o wide

Check the pod’s IP addresses:

kubectl get pods -n kube-system -l k8s-app=kube-dns -o jsonpath='{range .items[*]}{.metadata.name}: {.status.podIPs}{"\n"}{end}'

Each pod should have two IPs – one from 10.244.0.0/16 (IPv4) and one from fd00:10:244::/48 (IPv6).

Step 4: Configure CoreDNS for DNS64

With dual-stack pods, CoreDNS has IPv6 connectivity and can reach external DNS servers directly – no hostNetwork needed. However, CoreDNS must forward to DNS64 resolvers (not regular DNS) so that IPv4-only domains get synthesized AAAA records that pods can route to via NAT64.

4.1 Update CoreDNS ConfigMap

kubectl -n kube-system edit configmap coredns

Replace the forward line. Change:

forward . /etc/resolv.conf

To:

forward . 2001:67c:2b0::4 2001:67c:2b0::6

These are public DNS64 resolvers from nat64.net (Nuremberg and Helsinki – close to Hetzner’s datacenters). They synthesize AAAA records with the 64:ff9b::/96 prefix for IPv4-only domains.

Why DNS64 resolvers?

When a pod queries `api.anthropic.com`, CoreDNS forwards to the DNS64 resolver. If `api.anthropic.com` only has an A record (IPv4), the DNS64 resolver synthesizes an AAAA record like `64:ff9b::6812:0000`. The pod then sends IPv6 traffic to this address, Cilium masquerades it to the node's public IPv6, and the NAT64 gateway translates it to IPv4. This is how pods reach IPv4-only services without `hostNetwork`.

4.2 Restart CoreDNS

kubectl -n kube-system rollout restart deployment coredns
kubectl -n kube-system rollout status deployment coredns --timeout=60s

4.3 Verify DNS resolution

Create a long-running test pod (the --rm -it pattern tends to hang on IPv6-only clusters):

kubectl run test --image=alpine --restart=Never -- sleep 3600

Test DNS and connectivity:

# Test internal DNS (cluster service name)
kubectl exec test -- nslookup kubernetes.default.svc.cluster.local

# Test external DNS (should return a synthesized AAAA from the DNS64 resolver)
kubectl exec test -- nslookup github.com

# Install curl and test end-to-end NAT64 connectivity
kubectl exec test -- apk add --no-cache curl
kubectl exec test -- curl -6 -s --max-time 10 -o /dev/null -w "%{http_code}\n" https://github.com

Expected results: nslookup github.com returns both a synthesized AAAA (e.g. 2001:67c:2b0:db32:...) and a real A record. The curl -6 command forces IPv6 and should return 200 (or 301 for domains that redirect, like google.com).

BusyBox wget prefers IPv4

Alpine's BusyBox wget tries IPv4 first and has no `-6` flag. Since pods have no IPv4 internet route, `wget https://github.com` will fail with "Network unreachable". Use `curl -6` for testing, or note that glibc-based images (like `node:lts`) prefer IPv6 by default per RFC 6724.

Clean up:

kubectl delete pod test

Debugging DNS

If DNS resolution fails, check the CoreDNS logs: ```bash kubectl logs -n kube-system -l k8s-app=kube-dns -f ``` Verify that CoreDNS pods have IPv6 connectivity to the DNS64 resolvers: ```bash kubectl run test --image=alpine --restart=Never -- sleep 300 kubectl exec test -- ping6 -c 3 -W 3 2001:67c:2b0::4 kubectl delete pod test ```

Step 5: Install Hetzner CSI Driver

The Hetzner CSI driver enables persistent storage via Hetzner Block Volumes. With dual-stack networking, the CSI controller can reach api.hetzner.cloud through its pod IPv6 address and NAT64 – no hostNetwork patch needed.

5.1 Create a dedicated API token

In the Hetzner Cloud Console: Security > API Tokens > Generate API Token (Read & Write). Name it k8s-csi.

5.2 Deploy the CSI driver

# Create secret with API token
kubectl create secret generic hcloud \
  --namespace kube-system \
  --from-literal=token=<YOUR_HETZNER_CSI_API_TOKEN>

# Deploy CSI driver
kubectl apply -f https://raw.githubusercontent.com/hetznercloud/csi-driver/main/deploy/kubernetes/hcloud-csi.yml

# Set hcloud-volumes as default storage class
kubectl patch storageclass hcloud-volumes \
  -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

5.3 Verify

kubectl get storageclass
kubectl get pods -n kube-system | grep hcloud
# hcloud-csi-controller should show 5/5 Running
# hcloud-csi-node should show 3/3 Running

If the CSI controller fails to start with connection errors to api.hetzner.cloud, verify that:

CoreDNS is forwarding to DNS64 resolvers (Step 4)
The pod has an IPv6 address (kubectl get pods -n kube-system -o wide | grep hcloud-csi-controller)
The NAT64 route exists on the node (ip -6 route | grep 64:ff9b)

Step 6: Namespace Isolation and Network Policies

Use namespaces with Cilium network policies to isolate workloads and control egress traffic. With dual-stack networking, Cilium’s FQDN-based egress rules work for all pods – the DNS proxy intercepts DNS64-synthesized AAAA responses and allows traffic to those addresses.

6.1 Create namespaces

# namespaces.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: system-unrestricted
  labels:
    egress-policy: unrestricted
---
apiVersion: v1
kind: Namespace
metadata:
  name: apps-restricted
  labels:
    egress-policy: restricted

kubectl apply -f namespaces.yaml

system-unrestricted: For infrastructure services (cloudflared, monitoring) that need full network access.
apps-restricted: For application workloads with egress locked down to specific destinations.

6.2 Default deny egress for restricted namespace

# default-deny-egress.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-egress
  namespace: apps-restricted
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress: []

kubectl apply -f default-deny-egress.yaml

6.3 Whitelist specific egress with Cilium (optional)

This step is a reference example — apply it when deploying applications that need specific egress rules (see the OpenClaw guide for a real-world example).

Cilium supports FQDN-based egress rules, allowing fine-grained control over which external services an application can reach. With dual-stack, the DNS proxy intercepts DNS64-synthesized AAAA records and maps them to the FQDN, so toFQDNs rules work transparently with NAT64:

# example-app-egress.yaml
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: example-app-egress
  namespace: apps-restricted
spec:
  endpointSelector:
    matchLabels:
      app: my-app
  egress:
    # DNS resolution (required for FQDN rules)
    - toEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: kube-system
            k8s-app: kube-dns
      toPorts:
        - ports:
            - port: "53"
              protocol: UDP
            - port: "53"
              protocol: TCP

    # Allow internal cluster communication
    - toEntities:
        - cluster

    # Allow specific external API (example)
    - toFQDNs:
        - matchName: "api.example.com"
      toPorts:
        - ports:
            - port: "443"
              protocol: TCP

How FQDN rules work with DNS64/NAT64

When a pod queries `api.example.com`, Cilium's DNS proxy intercepts the response. If the DNS64 resolver returns a synthesized AAAA record (`64:ff9b::xxxx`), the proxy records the mapping `api.example.com → 64:ff9b::xxxx`. The `toFQDNs` rule then allows traffic to that synthesized address. From Cilium's perspective, a DNS64-synthesized AAAA record is just a regular AAAA record.

6.4 Unrestricted egress for system namespace

# system-unrestricted-egress.yaml
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: allow-all-egress
  namespace: system-unrestricted
spec:
  endpointSelector: {}
  egress:
    - toEntities:
        - all

kubectl apply -f system-unrestricted-egress.yaml

6.5 Verify network policies

Use wget (included in Alpine by default) to verify that the default-deny policy blocks all egress, including DNS resolution:

# Start a test pod in the restricted namespace
kubectl run test --namespace apps-restricted --rm -it --image=alpine -- sh

# Inside the pod -- both should FAIL with DNS or connection errors:
wget -qO- https://google.com
ping -c1 8.8.8.8

# Exit test pod
exit

Why not `curl`?

You might think to `apk add curl` first, but that itself requires DNS resolution and network access to reach the Alpine package mirror -- which is exactly what the default-deny egress policy blocks. The fact that `apk` fails is already proof that the policy works. Alpine ships with `wget`, so no package installation is needed.

Then verify that the unrestricted namespace allows full egress:

kubectl run test --namespace system-unrestricted --image=alpine --restart=Never -- sleep 3600

# Install curl and test (BusyBox wget lacks -6 and prefers IPv4 which is unreachable)
kubectl exec --namespace system-unrestricted test -- apk add --no-cache curl
kubectl exec --namespace system-unrestricted test -- curl -6 -s --max-time 10 -o /dev/null -w "%{http_code}\n" https://google.com

# Clean up
kubectl delete pod --namespace system-unrestricted test

The curl -6 command should return 301 (Google redirects to www.google.com), confirming full IPv6 egress works.

Step 7: Expose Services via Cloudflare Tunnel

The Cloudflare Tunnel (installed as a system service on the master control node via cloud-init) can route external traffic to Kubernetes services. This step deploys a test service in the restricted namespace to verify the full chain: Internet → Cloudflare → Tunnel → Kubernetes Service → Pod – with network policies enforced.

7.1 Deploy a test service

Apply the following manifest. It creates an Nginx deployment, a ClusterIP service, and a CiliumNetworkPolicy in the apps-restricted namespace. The network policy only allows DNS egress (which Nginx doesn’t strictly need, but demonstrates the pattern every real application requires):

# nginx-test.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-test
  namespace: apps-restricted
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-test
  template:
    metadata:
      labels:
        app: nginx-test
    spec:
      containers:
        - name: nginx
          image: nginx:alpine
          ports:
            - containerPort: 80
          resources:
            requests:
              memory: "32Mi"
              cpu: "10m"
            limits:
              memory: "64Mi"
              cpu: "100m"
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-test
  namespace: apps-restricted
spec:
  selector:
    app: nginx-test
  ports:
    - port: 80
      targetPort: 80
---
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: nginx-test-egress
  namespace: apps-restricted
spec:
  endpointSelector:
    matchLabels:
      app: nginx-test
  egress:
    # DNS resolution (required for most real applications)
    - toEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: kube-system
            k8s-app: kube-dns
      toPorts:
        - ports:
            - port: "53"
              protocol: UDP
            - port: "53"
              protocol: TCP

kubectl apply -f nginx-test.yaml

Verify it’s running:

kubectl get pods -n apps-restricted -l app=nginx-test
kubectl get svc nginx-test -n apps-restricted

7.2 Enable cluster DNS on the host

The cloudflared system service runs on the host, not inside the cluster. By default, it cannot resolve Kubernetes service names like nginx-test.apps-restricted.svc.cluster.local because the host uses Hetzner’s DNS servers, not CoreDNS.

Add CoreDNS as a nameserver on the host. CoreDNS listens on its pod IP (a ClusterIP or node-local address routable via Cilium):

# Find the CoreDNS ClusterIP
COREDNS_IP=$(kubectl get svc kube-dns -n kube-system -o jsonpath='{.spec.clusterIP}')
echo "CoreDNS ClusterIP: $COREDNS_IP"

sudo sed -i "1s/^/nameserver $COREDNS_IP\n/" /etc/resolv.conf

resolv.conf is managed by resolvconf

The `/etc/resolv.conf` file is auto-generated and may be overwritten on reboot or network changes. To make this permanent, add the nameserver to `/etc/resolvconf/resolv.conf.d/head`: ```bash echo "nameserver $COREDNS_IP" | sudo tee /etc/resolvconf/resolv.conf.d/head sudo resolvconf -u ```

Verify the host can resolve cluster service names:

curl http://nginx-test.apps-restricted.svc.cluster.local

You should see the Nginx welcome page.

7.3 Configure the tunnel hostname

In the Cloudflare dashboard:

Go to Zero Trust > Networks > Tunnels > your tunnel > Public Hostnames
Add a new public hostname:

Hostname	Service
`test.yourdomain.com`	`http://nginx-test.apps-restricted.svc.cluster.local:80`

SSL/TLS mode

In the Cloudflare dashboard under **yourdomain.com** > **SSL/TLS** > **Overview**, set the encryption mode to **Full** (not **Full (strict)**). The tunnel terminates TLS at Cloudflare and connects to the origin (Nginx) over plain HTTP.

7.4 Test access

From your local machine (or anywhere on the internet):

curl https://test.yourdomain.com

You should see the Nginx welcome page. This confirms the full chain works:

Internet → Cloudflare (TLS termination)
  → Tunnel → cloudflared (system service on host)
    → CoreDNS resolves service name → ClusterIP
      → Nginx pod (apps-restricted namespace, egress restricted by Cilium)

Ingress is allowed by default

The default-deny policy from Step 6.2 only restricts **egress**. Ingress to pods in `apps-restricted` is allowed, so cloudflared can reach Nginx without an additional ingress rule. For production workloads, consider adding explicit ingress policies as well.

7.5 Clean up

Remove the test resources and the Cloudflare public hostname:

kubectl delete -f nginx-test.yaml

Then in the Cloudflare dashboard: Zero Trust > Networks > Tunnels > your tunnel > Public Hostnames > delete the test.yourdomain.com entry.

Authentication

At this point, the tunnel exposes services without authentication. Adding Cloudflare Access policies (Zero Trust > Access > Applications) to control who can reach your services is covered in the [OpenClaw deployment guide](/kubeclaw/docs/guide/openclaw/).

7.6 Optional: Run cloudflared as a Kubernetes workload

The system-level cloudflared installed via cloud-init is sufficient for most setups. If you prefer to manage the tunnel as a Kubernetes deployment (for HA with multiple replicas, resource limits, and Kubernetes-native lifecycle management), you can migrate it:

# cloudflared.yaml
apiVersion: v1
kind: Secret
metadata:
  name: cloudflared-token
  namespace: system-unrestricted
type: Opaque
stringData:
  token: "<YOUR_CLOUDFLARE_TUNNEL_TOKEN>"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cloudflared
  namespace: system-unrestricted
spec:
  replicas: 2
  selector:
    matchLabels:
      app: cloudflared
  template:
    metadata:
      labels:
        app: cloudflared
    spec:
      containers:
        - name: cloudflared
          image: cloudflare/cloudflared:latest
          args:
            - tunnel
            - --no-autoupdate
            - run
            - --token
            - $(TUNNEL_TOKEN)
          env:
            - name: TUNNEL_TOKEN
              valueFrom:
                secretKeyRef:
                  name: cloudflared-token
                  key: token
          resources:
            requests:
              memory: "64Mi"
              cpu: "50m"
            limits:
              memory: "128Mi"
              cpu: "200m"

kubectl apply -f cloudflared.yaml

Then disable the system-level service:

ssh control-node sudo systemctl stop cloudflared
ssh control-node sudo systemctl disable cloudflared

Next Steps

For upgrade procedures and operational kubectl commands, see Kubernetes Maintenance.

4.6 - OpenClaw Deployment

Deploy OpenClaw into an egress-restricted namespace with Cilium FQDN policies.

A step-by-step guide to deploying OpenClaw as an isolated AI assistant on the KubeClaw cluster, accessible via Telegram, WhatsApp, or Signal, with a web-based Control UI exposed through Cloudflare Tunnel.

Architecture Overview

┌─────────────────────────────────────────────────────────────────────┐
│                    Hetzner Private Network                          │
│                        (10.0.0.0/24)                                │
│                                                                     │
│  ┌────────────────────────────────────────────────────────────┐    │
│  │            Kubernetes Cluster (kubeadm) + Cilium             │    │
│  │                                                             │    │
│  │   ┌─────────────────┐       ┌─────────────────┐            │    │
│  │   │   control-01    │       │   worker-01     │            │    │
│  │   │  (control plane)│◄─────►│  (worker node)  │            │    │
│  │   │   10.0.0.2      │       │   10.0.0.3      │            │    │
│  │   │                 │       │                 │            │    │
│  │   │  ┌───────────┐  │       │  ┌───────────┐  │            │    │
│  │   │  │Block Vol  │  │       │  │Block Vol  │  │            │    │
│  │   │  │  10 GB    │  │       │  │  10 GB    │  │            │    │
│  │   │  └───────────┘  │       │  └───────────┘  │            │    │
│  │   └────────┬────────┘       └─────────────────┘            │    │
│  │            │                                                │    │
│  │   system-unrestricted namespace:                           │    │
│  │     └─ cloudflared (egress: ANY)                           │    │
│  │                                                             │    │
│  │   apps-restricted namespace:                                │    │
│  │     └─ OpenClaw (Cilium FQDN egress whitelist)             │    │
│  │          ├─ Telegram / WhatsApp / Signal                   │    │
│  │          └─ Control UI (:18789)                            │    │
│  │                                                             │    │
│  └─────────────────────────────────────────────────────────────┘    │
│                              │                                      │
│                      cloudflared                                    │
│                    (outbound only)                                  │
└──────────────────────────────┼──────────────────────────────────────┘
                               │
                               ▼
                    ┌───────────────────┐
                    │    Cloudflare     │
                    │  Edge + Access    │
                    └─────────┬─────────┘
                              │
                    ┌─────────┼─────────┐
                    │         │         │
                    ▼         ▼         ▼
               [Telegram] [WhatsApp] [Signal]
                    │
                    ▼
             [Control UI]
          (browser dashboard)

Prerequisites

A running KubeClaw cluster (see Kubernetes guide)
Anthropic API key (console.anthropic.com)
At least one messaging channel:

Channel	What You Need
Telegram	Bot Token from @BotFather, your Telegram User ID
WhatsApp	A phone number with WhatsApp, access to scan a QR code
Signal	A dedicated phone number, `signal-cli` installed in the container

Start with Telegram

Telegram is the simplest channel to set up — it only requires a bot token and user ID, with no interactive pairing or additional dependencies.

Step 1: Infrastructure Setup

If you haven’t already set up the Hetzner Cloud infrastructure, follow the Quick Start and Kubernetes guide.

This guide assumes you have:

A running Kubernetes cluster with Cilium CNI
Hetzner CSI driver installed
Namespaces created (system-unrestricted, apps-restricted)
CoreDNS forwarding to DNS64 resolvers (see Kubernetes guide, Step 4)
Cloudflare Tunnel configured on the master control node

Step 2: Deploy OpenClaw

2.1 Create OpenClaw Configuration

OpenClaw uses JSON5 configuration (supports comments and trailing commas). Create a ConfigMap with the base configuration:

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: openclaw-config
  namespace: apps-restricted
data:
  openclaw.json: |
    {
      // Model configuration
      "agents": {
        "defaults": {
          "model": {
            // Primary model for conversations
            "primary": "anthropic/claude-sonnet-4-5-20250929",
            // Fallback chain: tried in order if the primary fails
            // (rate limit, auth error, timeout, outage)
            "fallbacks": [
              "anthropic/claude-haiku-4-5-20251001"
            ]
          },
          // Aliases appear in the Control UI model selector
          "models": {
            "anthropic/claude-sonnet-4-5-20250929": { "alias": "Sonnet" },
            "anthropic/claude-haiku-4-5-20251001": { "alias": "Haiku" },
            "anthropic/claude-opus-4-6": { "alias": "Opus" }
          }
        }
      },

      // Provider credentials
      "models": {
        "providers": {
          "anthropic": { "apiKey": "$ANTHROPIC_API_KEY" }
        }
      },

      // Messaging channel
      "channels": {
        "telegram": {
          "enabled": true,
          "dmPolicy": "allowlist",
          "allowFrom": ["<YOUR_TELEGRAM_USER_ID>"]
        }
      },

      // Gateway authentication (required for Control UI)
      "gateway": {
        "port": 18789,
        "bind": "lan",
        "auth": {
          "mode": "token"
        },
        "controlUi": {
          "enabled": true
        }
      }
    }
```

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: openclaw-config
  namespace: apps-restricted
data:
  openclaw.json: |
    {
      "agents": {
        "defaults": {
          "model": {
            "primary": "anthropic/claude-sonnet-4-5-20250929",
            "fallbacks": [
              "anthropic/claude-haiku-4-5-20251001"
            ]
          },
          "models": {
            "anthropic/claude-sonnet-4-5-20250929": { "alias": "Sonnet" },
            "anthropic/claude-haiku-4-5-20251001": { "alias": "Haiku" },
            "anthropic/claude-opus-4-6": { "alias": "Opus" }
          }
        }
      },

      "models": {
        "providers": {
          "anthropic": { "apiKey": "$ANTHROPIC_API_KEY" }
        }
      },

      "channels": {
        "whatsapp": {
          "enabled": true,
          "dmPolicy": "allowlist",
          "allowFrom": ["<YOUR_PHONE_E164>"]
        }
      },

      "gateway": {
        "port": 18789,
        "bind": "lan",
        "auth": {
          "mode": "token"
        },
        "controlUi": {
          "enabled": true
        }
      }
    }
```

<div class="alert alert-primary" role="alert"><div class="h4 alert-heading" role="heading">Phone number format</div>


Use E.164 format for `allowFrom`, e.g. `"+15551234567"` (with country code, no spaces).

</div>

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: openclaw-config
  namespace: apps-restricted
data:
  openclaw.json: |
    {
      "agents": {
        "defaults": {
          "model": {
            "primary": "anthropic/claude-sonnet-4-5-20250929",
            "fallbacks": [
              "anthropic/claude-haiku-4-5-20251001"
            ]
          },
          "models": {
            "anthropic/claude-sonnet-4-5-20250929": { "alias": "Sonnet" },
            "anthropic/claude-haiku-4-5-20251001": { "alias": "Haiku" },
            "anthropic/claude-opus-4-6": { "alias": "Opus" }
          }
        }
      },

      "models": {
        "providers": {
          "anthropic": { "apiKey": "$ANTHROPIC_API_KEY" }
        }
      },

      "channels": {
        "signal": {
          "enabled": true,
          "account": "<BOT_PHONE_E164>",
          "cliPath": "signal-cli",
          "dmPolicy": "allowlist",
          "allowFrom": ["<YOUR_PHONE_E164>"]
        }
      },

      "gateway": {
        "port": 18789,
        "bind": "lan",
        "auth": {
          "mode": "token"
        },
        "controlUi": {
          "enabled": true
        }
      }
    }
```

<div class="alert alert-primary" role="alert"><div class="h4 alert-heading" role="heading">Signal requires a custom container image</div>


Signal integration requires `signal-cli` (a Java application) installed in the container. See [Step 3: Signal Setup](#signal-setup) for details.

</div>

Replace placeholders:

<YOUR_TELEGRAM_USER_ID> — your numeric Telegram user ID (see Step 3: Telegram Setup)
<YOUR_PHONE_E164> — your phone number in E.164 format (e.g. +15551234567)
<BOT_PHONE_E164> — dedicated phone number for the Signal bot

Model configuration

The config above uses Sonnet as the primary model with Haiku as a fallback. OpenClaw automatically fails over when the primary model hits rate limits, auth errors, or timeouts. **Customizing models:** - **`model.primary`** — the default model for all conversations - **`model.fallbacks`** — ordered list of backup models, tried in sequence on failure - **`models`** — allowlist with aliases that appear in the Control UI model selector. You can switch models from the UI during a conversation - **`models.providers`** — provider credentials. Supports Anthropic, OpenAI, Google Gemini, OpenRouter, and local models (Ollama). Add multiple providers to mix models: ```json5 "models": { "providers": { "anthropic": { "apiKey": "$ANTHROPIC_API_KEY" }, "openai": { "apiKey": "$OPENAI_API_KEY" } } } ``` - **Per-agent overrides** — individual agents in `agents.list[]` can override the default model. See the [OpenClaw model docs](https://docs.openclaw.ai/concepts/models) for details.

2.2 Create Secrets

Generate a gateway authentication token and create the Kubernetes secret:

```bash
# Generate a strong gateway token
GATEWAY_TOKEN=$(openssl rand -hex 32)
echo "Save this token for Control UI access: $GATEWAY_TOKEN"

kubectl create secret generic openclaw-secrets \
  --namespace apps-restricted \
  --from-literal=ANTHROPIC_API_KEY=<YOUR_ANTHROPIC_API_KEY> \
  --from-literal=TELEGRAM_BOT_TOKEN=<YOUR_TELEGRAM_BOT_TOKEN> \
  --from-literal=OPENCLAW_GATEWAY_TOKEN=$GATEWAY_TOKEN
```

```bash
GATEWAY_TOKEN=$(openssl rand -hex 32)
echo "Save this token for Control UI access: $GATEWAY_TOKEN"

kubectl create secret generic openclaw-secrets \
  --namespace apps-restricted \
  --from-literal=ANTHROPIC_API_KEY=<YOUR_ANTHROPIC_API_KEY> \
  --from-literal=OPENCLAW_GATEWAY_TOKEN=$GATEWAY_TOKEN
```

<div class="alert alert-primary" role="alert"><div class="h4 alert-heading" role="heading">No API token needed</div>


WhatsApp uses QR-code based linking — no API keys or tokens required. You'll pair your account interactively in [Step 3](#whatsapp-setup).

</div>

```bash
GATEWAY_TOKEN=$(openssl rand -hex 32)
echo "Save this token for Control UI access: $GATEWAY_TOKEN"

kubectl create secret generic openclaw-secrets \
  --namespace apps-restricted \
  --from-literal=ANTHROPIC_API_KEY=<YOUR_ANTHROPIC_API_KEY> \
  --from-literal=OPENCLAW_GATEWAY_TOKEN=$GATEWAY_TOKEN
```

Save the gateway token

The `GATEWAY_TOKEN` is required to access the Control UI dashboard. Save it securely — you'll need it when logging into the web interface.

2.3 Deploy OpenClaw StatefulSet

Create file openclaw.yaml:

```yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: openclaw
  namespace: apps-restricted
spec:
  serviceName: openclaw
  replicas: 1
  selector:
    matchLabels:
      app: openclaw
  template:
    metadata:
      labels:
        app: openclaw
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        fsGroup: 1000
      containers:
        - name: openclaw
          image: node:22-slim
          workingDir: /app
          command:
            - /bin/sh
            - -c
            - |
              npm install -g openclaw@latest &&
              openclaw gateway --port 18789
          env:
            - name: ANTHROPIC_API_KEY
              valueFrom:
                secretKeyRef:
                  name: openclaw-secrets
                  key: ANTHROPIC_API_KEY
            - name: TELEGRAM_BOT_TOKEN
              valueFrom:
                secretKeyRef:
                  name: openclaw-secrets
                  key: TELEGRAM_BOT_TOKEN
            - name: OPENCLAW_GATEWAY_TOKEN
              valueFrom:
                secretKeyRef:
                  name: openclaw-secrets
                  key: OPENCLAW_GATEWAY_TOKEN
            - name: OPENCLAW_CONFIG_PATH
              value: /etc/openclaw/openclaw.json
            - name: OPENCLAW_STATE_DIR
              value: /home/node/.openclaw
          ports:
            - containerPort: 18789
              name: gateway
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
          resources:
            requests:
              memory: "512Mi"
              cpu: "250m"
            limits:
              memory: "2Gi"
              cpu: "2000m"
          volumeMounts:
            - name: config
              mountPath: /etc/openclaw/openclaw.json
              subPath: openclaw.json
              readOnly: true
            - name: data
              mountPath: /home/node/.openclaw
      volumes:
        - name: config
          configMap:
            name: openclaw-config
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes:
          - ReadWriteOnce
        storageClassName: hcloud-volumes
        resources:
          requests:
            storage: 10Gi
---
apiVersion: v1
kind: Service
metadata:
  name: openclaw
  namespace: apps-restricted
spec:
  selector:
    app: openclaw
  ports:
    - port: 18789
      targetPort: 18789
      name: gateway
---
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: openclaw-egress
  namespace: apps-restricted
spec:
  endpointSelector:
    matchLabels:
      app: openclaw
  egress:
    # DNS resolution (required for FQDN rules)
    - toEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: kube-system
            k8s-app: kube-dns
      toPorts:
        - ports:
            - port: "53"
              protocol: UDP
            - port: "53"
              protocol: TCP

    # Allow internal cluster communication
    - toEntities:
        - cluster

    # Anthropic API
    - toFQDNs:
        - matchName: "api.anthropic.com"
      toPorts:
        - ports:
            - port: "443"
              protocol: TCP

    # Telegram API
    - toFQDNs:
        - matchName: "api.telegram.org"
      toPorts:
        - ports:
            - port: "443"
              protocol: TCP
```

```yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: openclaw
  namespace: apps-restricted
spec:
  serviceName: openclaw
  replicas: 1
  selector:
    matchLabels:
      app: openclaw
  template:
    metadata:
      labels:
        app: openclaw
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        fsGroup: 1000
      containers:
        - name: openclaw
          image: node:22-slim
          workingDir: /app
          command:
            - /bin/sh
            - -c
            - |
              npm install -g openclaw@latest &&
              openclaw gateway --port 18789
          env:
            - name: ANTHROPIC_API_KEY
              valueFrom:
                secretKeyRef:
                  name: openclaw-secrets
                  key: ANTHROPIC_API_KEY
            - name: OPENCLAW_GATEWAY_TOKEN
              valueFrom:
                secretKeyRef:
                  name: openclaw-secrets
                  key: OPENCLAW_GATEWAY_TOKEN
            - name: OPENCLAW_CONFIG_PATH
              value: /etc/openclaw/openclaw.json
            - name: OPENCLAW_STATE_DIR
              value: /home/node/.openclaw
          ports:
            - containerPort: 18789
              name: gateway
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
          resources:
            requests:
              memory: "512Mi"
              cpu: "250m"
            limits:
              memory: "2Gi"
              cpu: "2000m"
          volumeMounts:
            - name: config
              mountPath: /etc/openclaw/openclaw.json
              subPath: openclaw.json
              readOnly: true
            - name: data
              mountPath: /home/node/.openclaw
      volumes:
        - name: config
          configMap:
            name: openclaw-config
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes:
          - ReadWriteOnce
        storageClassName: hcloud-volumes
        resources:
          requests:
            storage: 10Gi
---
apiVersion: v1
kind: Service
metadata:
  name: openclaw
  namespace: apps-restricted
spec:
  selector:
    app: openclaw
  ports:
    - port: 18789
      targetPort: 18789
      name: gateway
---
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: openclaw-egress
  namespace: apps-restricted
spec:
  endpointSelector:
    matchLabels:
      app: openclaw
  egress:
    # DNS resolution (required for FQDN rules)
    - toEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: kube-system
            k8s-app: kube-dns
      toPorts:
        - ports:
            - port: "53"
              protocol: UDP
            - port: "53"
              protocol: TCP

    # Allow internal cluster communication
    - toEntities:
        - cluster

    # Anthropic API
    - toFQDNs:
        - matchName: "api.anthropic.com"
      toPorts:
        - ports:
            - port: "443"
              protocol: TCP

    # WhatsApp servers
    - toFQDNs:
        - matchName: "web.whatsapp.com"
        - matchPattern: "*.whatsapp.net"
        - matchPattern: "*.whatsapp.com"
      toPorts:
        - ports:
            - port: "443"
              protocol: TCP
            - port: "5222"
              protocol: TCP
```

```yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: openclaw
  namespace: apps-restricted
spec:
  serviceName: openclaw
  replicas: 1
  selector:
    matchLabels:
      app: openclaw
  template:
    metadata:
      labels:
        app: openclaw
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        fsGroup: 1000
      containers:
        - name: openclaw
          image: your-registry/openclaw-signal:latest  # Custom image with signal-cli
          env:
            - name: ANTHROPIC_API_KEY
              valueFrom:
                secretKeyRef:
                  name: openclaw-secrets
                  key: ANTHROPIC_API_KEY
            - name: OPENCLAW_GATEWAY_TOKEN
              valueFrom:
                secretKeyRef:
                  name: openclaw-secrets
                  key: OPENCLAW_GATEWAY_TOKEN
            - name: OPENCLAW_CONFIG_PATH
              value: /etc/openclaw/openclaw.json
            - name: OPENCLAW_STATE_DIR
              value: /home/node/.openclaw
          ports:
            - containerPort: 18789
              name: gateway
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
          resources:
            requests:
              memory: "512Mi"
              cpu: "250m"
            limits:
              memory: "2Gi"
              cpu: "2000m"
          volumeMounts:
            - name: config
              mountPath: /etc/openclaw/openclaw.json
              subPath: openclaw.json
              readOnly: true
            - name: data
              mountPath: /home/node/.openclaw
      volumes:
        - name: config
          configMap:
            name: openclaw-config
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes:
          - ReadWriteOnce
        storageClassName: hcloud-volumes
        resources:
          requests:
            storage: 10Gi
---
apiVersion: v1
kind: Service
metadata:
  name: openclaw
  namespace: apps-restricted
spec:
  selector:
    app: openclaw
  ports:
    - port: 18789
      targetPort: 18789
      name: gateway
---
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: openclaw-egress
  namespace: apps-restricted
spec:
  endpointSelector:
    matchLabels:
      app: openclaw
  egress:
    # DNS resolution (required for FQDN rules)
    - toEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: kube-system
            k8s-app: kube-dns
      toPorts:
        - ports:
            - port: "53"
              protocol: UDP
            - port: "53"
              protocol: TCP

    # Allow internal cluster communication
    - toEntities:
        - cluster

    # Anthropic API
    - toFQDNs:
        - matchName: "api.anthropic.com"
      toPorts:
        - ports:
            - port: "443"
              protocol: TCP

    # Signal servers
    - toFQDNs:
        - matchName: "chat.signal.org"
        - matchName: "storage.signal.org"
        - matchName: "cdn.signal.org"
        - matchName: "cdn2.signal.org"
        - matchName: "contentproxy.signal.org"
      toPorts:
        - ports:
            - port: "443"
              protocol: TCP
```

Cilium FQDN egress whitelist

Each manifest includes a `CiliumNetworkPolicy` that restricts OpenClaw's outbound traffic to only the Anthropic API and the messaging provider's servers. Combined with the namespace-level default-deny policy from the [Kubernetes guide](/kubeclaw/docs/guide/kubernetes/), this means OpenClaw **cannot** reach any other external service. Cilium's DNS proxy intercepts DNS64-synthesized AAAA responses and maps them to the FQDN, so these rules work transparently with NAT64.

Startup time

The pod installs OpenClaw via `npm install -g` on every restart, which takes 1–2 minutes. For faster restarts, build a custom Docker image with OpenClaw baked in (see [Maintenance](#build-a-custom-image-optional)).

Apply:

kubectl apply -f openclaw-config.yaml
kubectl apply -f openclaw.yaml

2.4 Verify Deployment

# Watch pod status (wait for Running)
kubectl get pods -n apps-restricted -w

# Check logs (wait for "gateway listening" message)
kubectl logs -n apps-restricted -l app=openclaw -f

# Check persistent volume
kubectl get pvc -n apps-restricted

The gateway is ready when you see output like:

gateway listening on 0.0.0.0:18789

Step 3: Configure Messaging Channel

Telegram Setup

3.1 Create a Bot

Open Telegram and message @BotFather
Send /newbot and follow the prompts
Save the bot token (format: 123456789:ABCdefGHI...)

3.2 Find Your User ID

The safest way (no third-party bots required):

Message your new bot (send any message)
Check the OpenClaw logs:

kubectl logs -n apps-restricted -l app=openclaw -f

Look for your numeric ID in the from.id field of the incoming message log

Alternatively, query the Telegram Bot API directly:

curl "https://api.telegram.org/bot<BOT_TOKEN>/getUpdates"

3.3 Update Configuration

Edit the ConfigMap with your actual user ID:

kubectl edit configmap openclaw-config -n apps-restricted

Replace <YOUR_TELEGRAM_USER_ID> with your numeric ID, then restart:

kubectl rollout restart statefulset/openclaw -n apps-restricted

3.4 Test

Open Telegram and find your bot
Send any message
If using dmPolicy: "pairing", approve the pairing request:

kubectl exec -n apps-restricted -it openclaw-0 -- openclaw pairing list telegram
kubectl exec -n apps-restricted -it openclaw-0 -- openclaw pairing approve telegram <CODE>

If using dmPolicy: "allowlist" (recommended), your user ID is pre-approved and the bot responds immediately.

WhatsApp Setup

3.1 Link Your WhatsApp Account

WhatsApp uses QR-code based pairing — no API keys needed. Exec into the pod and run the login command:

kubectl exec -n apps-restricted -it openclaw-0 -- openclaw channels login --channel whatsapp

This displays a QR code in the terminal. Scan it with your WhatsApp app:

Open WhatsApp on your phone
Go to Settings > Linked Devices > Link a Device
Scan the QR code

Dedicated number recommended

Using a separate WhatsApp number (not your personal one) provides cleaner access boundaries and avoids self-chat confusion. If using your personal number, add `"selfChatMode": true` to the WhatsApp channel configuration.

3.2 Verify Connection

After linking, restart the gateway to pick up the stored credentials:

kubectl rollout restart statefulset/openclaw -n apps-restricted

Check logs for WhatsApp connection:

kubectl logs -n apps-restricted -l app=openclaw -f

3.3 Network Policy Considerations

WhatsApp (via the Baileys library) connects to multiple WhatsApp servers. The key domains include:

web.whatsapp.com
*.whatsapp.net (messaging, media)

WhatsApp credentials persist
WhatsApp credentials are stored at `~/.openclaw/credentials/whatsapp/` on the persistent volume. They survive pod restarts — you only need to scan the QR code once.

Signal Setup

Advanced setup

Signal requires `signal-cli`, a Java-based application, installed in the container. This section requires building a custom Docker image.

3.1 Build a Custom Image with signal-cli

Create a Dockerfile.openclaw-signal:

FROM node:22-slim

# Install signal-cli native build
RUN apt-get update && apt-get install -y curl && \
    VERSION=$(curl -Ls -o /dev/null -w '%{url_effective}' \
      https://github.com/AsamK/signal-cli/releases/latest | \
      sed -e 's/^.*\/v//') && \
    curl -L -O "https://github.com/AsamK/signal-cli/releases/download/v${VERSION}/signal-cli-${VERSION}-Linux-native.tar.gz" && \
    tar xf "signal-cli-${VERSION}-Linux-native.tar.gz" -C /opt && \
    ln -sf /opt/signal-cli /usr/local/bin/signal-cli && \
    rm -f "signal-cli-${VERSION}-Linux-native.tar.gz" && \
    apt-get remove -y curl && apt-get autoremove -y && \
    rm -rf /var/lib/apt/lists/*

# Install OpenClaw globally
RUN npm install -g openclaw@latest

USER 1000
WORKDIR /app

CMD ["openclaw", "gateway", "--port", "18789"]

Build and push to a registry accessible from your cluster, then update the StatefulSet image field.

3.2 Register the Bot Phone Number

Exec into the pod and register:

kubectl exec -n apps-restricted -it openclaw-0 -- sh

# Register the bot number (may require captcha)
signal-cli -a +<BOT_PHONE_NUMBER> register

# If captcha required, visit https://signalcaptchas.org/registration/generate.html
# then run:
signal-cli -a +<BOT_PHONE_NUMBER> register --captcha '<CAPTCHA_URL>'

# Verify with the SMS code
signal-cli -a +<BOT_PHONE_NUMBER> verify <CODE>

3.3 Alternatively: Link an Existing Account

If you prefer to link to an existing Signal account (not recommended for production):

kubectl exec -n apps-restricted -it openclaw-0 -- signal-cli link -n "OpenClaw"

Scan the QR code in your Signal app under Settings > Linked Devices.

Registering deauthenticates

Registering a phone number with `signal-cli` deauthenticates the main Signal app for that number. Use a **dedicated bot number** to avoid losing access to your personal Signal account.

Step 4: Expose Control UI via Cloudflare Tunnel

OpenClaw includes a web-based Control UI (dashboard) for managing sessions, viewing logs, and chatting directly. Since the dashboard is an admin surface, we protect it with two layers: Cloudflare Access (identity verification) and gateway token (application authentication).

4.1 Add Tunnel Route

In Cloudflare Zero Trust > Networks > Tunnels > your tunnel > Public Hostnames, add:

Hostname	Service
`openclaw.yourdomain.com`	`http://openclaw.apps-restricted.svc.cluster.local:18789`

SSL/TLS settings

In Cloudflare dashboard under your domain > **SSL/TLS** > **Overview**, set the mode to **Full** (not "Full (strict)") since the origin (OpenClaw gateway) serves HTTP, not HTTPS. Cloudflare terminates TLS at the edge.

4.2 Create Cloudflare Access Policy

Protect the dashboard route so only authenticated users can reach it:

Go to Cloudflare Zero Trust > Access > Applications
Click Add an Application > Self-hosted
Configure:

Field	Value
Application name	OpenClaw Dashboard
Session Duration	24 hours
Application domain	`openclaw.yourdomain.com`

Add a Policy:

Field	Value
Policy name	Allowed Users
Action	Allow
Include	Emails — `your-email@example.com`

Under Authentication, select your identity provider or use One-time PIN (sends a verification code to your email — no IdP setup required)

Two-layer authentication

With this setup, accessing `openclaw.yourdomain.com` requires: 1. **Cloudflare Access**: Verifies your identity (email + OTP or IdP login) 2. **Gateway token**: Entered in the Control UI settings panel (the token from [Step 2.2](#22-create-secrets)) Even if someone bypasses Cloudflare Access, they cannot use the dashboard without the gateway token.

4.3 Access the Dashboard

Navigate to https://openclaw.yourdomain.com in your browser
Authenticate with Cloudflare Access (email OTP or your identity provider)
The Control UI loads — enter the gateway token when prompted
The token is stored in your browser’s localStorage for future sessions

The Control UI provides:

Chat: Direct conversation with the AI agent
Sessions: View and manage active sessions across all channels
Channels: Status of connected messaging channels (Telegram, WhatsApp, Signal)
Logs: Live gateway log tailing
Configuration: Edit settings with concurrent edit protection
Skills: Install and manage skills

Step 5: Verify & Test

Test Messaging

1. Find your bot on Telegram
2. Send any message
3. OpenClaw should respond (only to your user ID)

1. Send a message to the linked WhatsApp number
2. OpenClaw should respond (only to numbers in `allowFrom`)

1. Send a message to the bot's Signal number
2. OpenClaw should respond (only to numbers in `allowFrom`)

Test Control UI

Open https://openclaw.yourdomain.com
Verify Cloudflare Access prompts for authentication
Enter the gateway token in the UI
Send a test message in the chat interface

Check Pod Health

# Pod status
kubectl get pods -n apps-restricted

# Logs
kubectl logs -n apps-restricted -l app=openclaw -f

# PVC status
kubectl get pvc -n apps-restricted

# Exec into pod for diagnostics
kubectl exec -n apps-restricted -it openclaw-0 -- openclaw doctor
kubectl exec -n apps-restricted -it openclaw-0 -- openclaw status

Maintenance

Update OpenClaw

The pod installs openclaw@latest on every restart, so a simple restart pulls the newest version:

kubectl rollout restart statefulset/openclaw -n apps-restricted

Rotate Secrets

# Generate new gateway token
NEW_GATEWAY_TOKEN=$(openssl rand -hex 32)
echo "New gateway token: $NEW_GATEWAY_TOKEN"

# Delete and recreate the secret (adjust for your channel)
kubectl delete secret openclaw-secrets -n apps-restricted
kubectl create secret generic openclaw-secrets \
  --namespace apps-restricted \
  --from-literal=ANTHROPIC_API_KEY=<YOUR_KEY> \
  --from-literal=TELEGRAM_BOT_TOKEN=<YOUR_TOKEN> \
  --from-literal=OPENCLAW_GATEWAY_TOKEN=$NEW_GATEWAY_TOKEN

# Restart to pick up new secrets
kubectl rollout restart statefulset/openclaw -n apps-restricted

Update your browser

After rotating the gateway token, you'll need to enter the new token in the Control UI settings panel.

Build a Custom Image (Optional)

For faster pod restarts (skipping npm install on every boot), build a custom image:

FROM node:22-slim
RUN npm install -g openclaw@latest
USER 1000
WORKDIR /app
CMD ["openclaw", "gateway", "--port", "18789"]

Pin the version

In production, pin to a specific version (`openclaw@1.2.3`) rather than `@latest` to avoid unexpected breaking changes on restart.

Useful Commands

# OpenClaw logs
kubectl logs -n apps-restricted -l app=openclaw -f

# Restart OpenClaw
kubectl rollout restart statefulset/openclaw -n apps-restricted

# Run diagnostics
kubectl exec -n apps-restricted -it openclaw-0 -- openclaw doctor

# Check channel status
kubectl exec -n apps-restricted -it openclaw-0 -- openclaw channels status --probe

# Interactive shell
kubectl exec -n apps-restricted -it openclaw-0 -- sh

# Check Cilium endpoint status (from control node)
cilium endpoint list

# Check FQDN cache (from control node)
kubectl exec -n kube-system -it \
  $(kubectl get pods -n kube-system -l k8s-app=cilium -o name | head -1) \
  -- cilium fqdn cache list

Configuration Changes

To update the OpenClaw configuration:

# Edit the ConfigMap
kubectl edit configmap openclaw-config -n apps-restricted

# Restart to apply
kubectl rollout restart statefulset/openclaw -n apps-restricted

Alternatively, update the YAML file and reapply:

kubectl apply -f openclaw-config.yaml
kubectl rollout restart statefulset/openclaw -n apps-restricted

5 - How-to

Focused procedures for common KubeClaw tasks.

Practical recipes for managing SSH keys, plus a manual walkthrough of the infrastructure OpenTofu builds for you.

5.1 - SSH Keys with Passphrase

Protect cluster SSH keys with a passphrase and keep Ansible working through ssh-agent.

Why use a passphrase?

An SSH key without a passphrase is like a house key without a lock on the key cabinet. If your laptop or key file is stolen, the attacker gains immediate access to your cluster.

Creating keys with a passphrase

# Control node key
ssh-keygen -t ed25519 -f ~/.ssh/k8s-cluster_control-node_key -C "k8s-control"
# Enter a strong passphrase when prompted

# Worker node key
ssh-keygen -t ed25519 -f ~/.ssh/k8s-cluster_worker-node_key -C "k8s-worker"
# Enter a passphrase when prompted

Using ssh-agent

Since Terraform and Ansible cannot directly use encrypted keys, you must use ssh-agent:

# Start agent (if not already running)
eval "$(ssh-agent -s)"

# Add keys (prompts for passphrase)
ssh-add ~/.ssh/k8s-cluster_control-node_key
ssh-add ~/.ssh/k8s-cluster_worker-node_key

# Check which keys are loaded
ssh-add -l

macOS: Keychain integration

On macOS you can store the passphrase in the system Keychain so the key is automatically available after a reboot:

# Add key AND store passphrase in Keychain
ssh-add --apple-use-keychain ~/.ssh/k8s-cluster_control-node_key
ssh-add --apple-use-keychain ~/.ssh/k8s-cluster_worker-node_key

Also add the following to ~/.ssh/config:

Host *
    UseKeychain yes
    AddKeysToAgent yes

Helper script

The project includes a script that sets up ssh-agent correctly:

# Run once before using SSH/Ansible
source ./scripts/ssh-agent-setup.sh

# Afterwards SSH and Ansible work without further passphrase prompts
ssh control-node
ansible all -m ping

Encrypting auto-generated keys after export

If you use auto-generated keys from OpenTofu, you can add a passphrase afterwards:

# Export key (unencrypted from state)
tofu output -raw control_node_ssh_private_key > ~/.ssh/k8s-cluster_control-node_key
chmod 600 ~/.ssh/k8s-cluster_control-node_key

# Add passphrase
ssh-keygen -p -f ~/.ssh/k8s-cluster_control-node_key
# Old passphrase: [Enter] (empty)
# New passphrase: [enter passphrase]
# Confirm: [repeat]

Note

After encryption, `tofu output` still returns the unencrypted key from the state. However, your local key file is now protected.

5.2 - Store SSH Keys in Password Manager

Back up and restore cluster SSH keys using a password manager.

Storing SSH keys in a password manager provides a secure backup that survives hardware failures and makes it easy to restore access from a new machine.

General workflow

This workflow applies to any password manager that supports secure notes or file attachments (Dashlane, 1Password, Bitwarden, etc.).

1. Create keys locally

ssh-keygen -t ed25519 -f ~/.ssh/k8s-cluster_control-node_key -C "control-node"
ssh-keygen -t ed25519 -f ~/.ssh/k8s-cluster_worker-node_key -C "worker-node"

2. Save to your password manager

Create a secure note or vault entry:

Name: “K8s Cluster SSH Keys” (or similar)
Private key: paste the contents of ~/.ssh/k8s-cluster_control-node_key
Public key: paste the contents of ~/.ssh/k8s-cluster_control-node_key.pub

Tip

Most password managers support file attachments -- you can attach the key files directly instead of pasting.

3. Add public key to terraform.tfvars

control_node_public_key = "ssh-ed25519 AAAA... control-node"
worker_node_public_key  = "ssh-ed25519 AAAA... worker-node"

4. Restore when needed

When setting up on a new machine:

Copy the private key from your password manager
Save it to ~/.ssh/k8s-cluster_control-node_key
Fix permissions: chmod 600 ~/.ssh/k8s-cluster_control-node_key

Example: Dashlane

Create a Secure Note in Dashlane
Name: “K8s Cluster SSH Keys”
Content: paste the private key (cat ~/.ssh/k8s-cluster_control-node_key)
Add the public key as an additional field
Optionally attach the key files directly to the secure note

5.3 - Manual Setup (Alternative)

Build the same infrastructure by hand in the Hetzner Console – the steps OpenTofu automates.

This guide is an alternative to OpenTofu

This guide shows the manual steps that OpenTofu automates. Use this if you want to understand what happens behind the scenes, or if you prefer to set up infrastructure manually via the Hetzner Cloud Console. For the automated approach, see [Infrastructure (OpenTofu)](/kubeclaw/docs/guide/infrastructure/).

Overview

This setup creates a secure server infrastructure with the following properties:

No public IPv4/IPv6 addresses (after setup)
SSH access exclusively via Cloudflare Tunnel
Internal communication via Hetzner Private Network
Hardened SSH configuration with fail2ban and UFW

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                         Internet                                 │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│                    Cloudflare Tunnel                             │
│                 console.yourdomain.org                           │
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│                    Hetzner Cloud                                 │
│  ┌─────────────────────────────────────────────────────────┐    │
│  │              Private Network (10.0.0.0/24)              │    │
│  │                                                         │    │
│  │   ┌─────────────────┐       ┌─────────────────┐        │    │
│  │   │  control-node   │       │  worker-node    │        │    │
│  │   │    10.0.0.2     │◄─────►│    10.0.0.3     │        │    │
│  │   │  (cloudflared)  │       │  (isolated)     │        │    │
│  │   └─────────────────┘       └─────────────────┘        │    │
│  │                                                         │    │
│  └─────────────────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────────────────┘

Prerequisites

Hetzner Cloud account
Cloudflare account with your own domain
macOS/Linux machine with SSH
cloudflared installed locally (brew install cloudflared)

Step 1: Create Hetzner Private Network

Open the Hetzner Cloud Console
Select your project
Go to Networks > Create Network
Configure:
- Name: k8s-network (or any name)
- IP Range: 10.0.0.0/8 (Hetzner requires /8 for the network object)
Click Create Network
Add a Subnet: 10.0.0.0/24 in zone eu-central (this is the actual range used by nodes)

Step 2: Generate SSH keys

Create a separate SSH key for each server:

# Control Node Key
ssh-keygen -t ed25519 -f ~/.ssh/k8s-cluster_control-node_key -C "control-node"

# Worker Node Key
ssh-keygen -t ed25519 -f ~/.ssh/k8s-cluster_worker-node_key -C "worker-node"

# Temporary Admin Node Key (for initial setup)
ssh-keygen -t ed25519 -f ~/.ssh/k8s-cluster_admin-node_key -C "admin-node"

Step 3: Create temporary Admin Node

Since the Hetzner Web Console (VNC) has issues with copy/paste (especially on Firefox/macOS), create a temporary admin server with a public IPv6 address for the initial setup.

Cloud-Init for Admin Node

#cloud-config

users:
  - name: kubernetes-admin
    groups: users, admin, sudo
    sudo: ALL=(ALL) NOPASSWD:ALL
    shell: /bin/bash
    ssh_authorized_keys:
      - ssh-ed25519 AAAA... YOUR_ADMIN_NODE_PUBLIC_KEY

keyboard:
  layout: de
  variant: mac

packages:
  - fail2ban
  - ufw

package_update: true
package_upgrade: true

write_files:
  - path: /etc/ssh/sshd_config.d/ssh-hardening.conf
    content: |
      PermitRootLogin no
      PasswordAuthentication no
      KbdInteractiveAuthentication no
      ChallengeResponseAuthentication no
      MaxAuthTries 3
      AllowTcpForwarding yes
      X11Forwarding no
      AllowAgentForwarding no
      AllowUsers kubernetes-admin
      ClientAliveInterval 300
      ClientAliveCountMax 2

  - path: /etc/fail2ban/jail.local
    content: |
      [sshd]
      enabled = true
      port = 22
      banaction = iptables-multiport
      maxretry = 3
      findtime = 600
      bantime = 3600

runcmd:
  - systemctl enable fail2ban
  - systemctl start fail2ban
  - ufw allow 22
  - ufw --force enable
  - reboot

Create the server

Servers > Add Server
Location: Any (e.g. Falkenstein)
Image: Debian 13
Type: CX23 (smallest size is sufficient)
Networking:
- Public IPv6 enabled
- Private Network: add your network
SSH Keys: Add Admin Node key
Cloud config: Paste the YAML above
Create & Buy now

Determine IPv6 address

The IPv6 address is displayed in Hetzner only as a subnet (e.g. 2a01:4f8:1c19:c886::/64). The actual server address is typically ::1 appended:

2a01:4f8:1c19:c886::1

Test connection

ssh -i ~/.ssh/k8s-cluster_admin-node_key kubernetes-admin@2a01:4f8:1c19:c886::1

Note

Your local machine needs IPv6 connectivity. Test with `ping6 google.com`.

Step 4: Create Control Node (with Cloudflare Tunnel)

Cloud-Init for Control Node

#cloud-config

users:
  - name: root
    plain_text_passwd: 'SECURE_PASSWORD_HERE'
    lock_passwd: false
  - name: kubernetes-admin
    groups: users, admin, sudo
    sudo: ALL=(ALL) NOPASSWD:ALL
    shell: /bin/bash
    ssh_authorized_keys:
      - ssh-ed25519 AAAA... YOUR_CONTROL_NODE_PUBLIC_KEY

keyboard:
  layout: de
  variant: mac

packages:
  - fail2ban
  - ufw
  - curl
  - wget

package_update: true
package_upgrade: true

write_files:
  - path: /etc/ssh/sshd_config.d/ssh-hardening.conf
    content: |
      PermitRootLogin no
      PasswordAuthentication no
      KbdInteractiveAuthentication no
      ChallengeResponseAuthentication no
      MaxAuthTries 3
      X11Forwarding no
      AllowAgentForwarding no
      AllowTcpForwarding yes
      AllowUsers kubernetes-admin
      ClientAliveInterval 300
      ClientAliveCountMax 2

  - path: /etc/fail2ban/jail.local
    content: |
      [sshd]
      enabled = true
      port = 22
      banaction = iptables-multiport
      maxretry = 3
      findtime = 600
      bantime = 3600

  - path: /etc/cloudflared/config.yml
    content: |
      edge-ip-version: "6"

runcmd:
  - systemctl enable fail2ban
  - systemctl start fail2ban
  - ufw allow from 10.0.0.0/24 to any port 22 proto tcp comment 'SSH internal'
  - ufw allow from 127.0.0.1 to any port 22 proto tcp comment 'SSH via Tunnel'
  - ufw default deny incoming
  - ufw default allow outgoing
  - ufw --force enable
  - mkdir -p --mode=0755 /usr/share/keyrings
  - curl -fsSL https://pkg.cloudflare.com/cloudflare-public-v2.gpg | tee /usr/share/keyrings/cloudflare-public-v2.gpg >/dev/null
  - echo 'deb [signed-by=/usr/share/keyrings/cloudflare-public-v2.gpg] https://pkg.cloudflare.com/cloudflared any main' | tee /etc/apt/sources.list.d/cloudflared.list
  - mkdir -p /etc/cloudflared
  - apt-get update && apt-get install -y cloudflared
  - reboot

Create the server

Servers > Add Server
Image: Debian 13
Type: As needed (e.g. CX23 or larger)
Networking:
- Public IPv4: Disabled
- Public IPv6: Enabled (temporarily, for installation)
- Private Network: add your network
SSH Keys: Add Control Node key
Cloud config: Paste the YAML above
Create & Buy now

Important

The `edge-ip-version: "6"` setting is essential! Cloudflared attempts IPv4 connections to Cloudflare's edge servers by default. Since the server is IPv6-only, this must be set to `"6"` (as a string!).

Step 5: Create Worker Node (isolated)

Cloud-Init for Worker Node

#cloud-config

users:
  - name: root
    plain_text_passwd: 'SECURE_PASSWORD_HERE'
    lock_passwd: false
  - name: kubernetes-admin
    groups: users, admin, sudo
    sudo: ALL=(ALL) NOPASSWD:ALL
    shell: /bin/bash
    ssh_authorized_keys:
      - ssh-ed25519 AAAA... YOUR_WORKER_NODE_PUBLIC_KEY

keyboard:
  layout: de
  variant: mac

packages:
  - fail2ban
  - ufw

package_update: true
package_upgrade: true

write_files:
  - path: /etc/ssh/sshd_config.d/ssh-hardening.conf
    content: |
      PermitRootLogin no
      PasswordAuthentication no
      KbdInteractiveAuthentication no
      ChallengeResponseAuthentication no
      MaxAuthTries 3
      AllowTcpForwarding no
      X11Forwarding no
      AllowAgentForwarding no
      AllowUsers kubernetes-admin
      ClientAliveInterval 300
      ClientAliveCountMax 2

  - path: /etc/fail2ban/jail.local
    content: |
      [sshd]
      enabled = true
      port = 22
      banaction = iptables-multiport
      maxretry = 3
      findtime = 600
      bantime = 3600

runcmd:
  - systemctl enable fail2ban
  - systemctl start fail2ban
  - ufw default deny incoming
  - ufw default deny outgoing
  - ufw allow from 10.0.0.0/24 to any port 22 proto tcp comment 'SSH internal'
  - ufw allow from 10.0.0.0/24 proto icmp comment 'ICMP internal'
  - ufw allow out to 10.0.0.0/24 comment 'Outbound internal'
  - ufw allow out to 185.12.64.1 port 53 proto udp comment 'DNS Hetzner'
  - ufw allow out to 185.12.64.2 port 53 proto udp comment 'DNS Hetzner'
  - ufw allow out to any port 80 proto tcp comment 'HTTP Updates'
  - ufw allow out to any port 443 proto tcp comment 'HTTPS Updates'
  - ufw --force enable
  - reboot

Create the server

Servers > Add Server
Image: Debian 13
Type: As needed
Networking:
- Public IPv4: Disabled
- Public IPv6: Enabled (temporarily)
- Private Network: add your network
SSH Keys: Add Worker Node key
Cloud config: Paste the YAML above
Create & Buy now

Step 6: Set up Cloudflare Tunnel

6.1 Create tunnel in Cloudflare

Open the Cloudflare Zero Trust Dashboard
Go to Networks > Tunnels
Click Create a tunnel
Select Cloudflared as connector
Tunnel name: e.g. hetzner-cluster
Save tunnel
Copy the install token (needed in the next step)

6.2 Install cloudflared on Control Node

Connect to the Control Node via the Admin Node:

# First connect to Admin Node
ssh admin-node

# Then connect to Control Node (via internal network)
ssh kubernetes-admin@10.0.0.2

On the Control Node:

# Install tunnel with token
sudo cloudflared service install <YOUR_TUNNEL_TOKEN>

# Check status
sudo systemctl status cloudflared

The tunnel should now appear as “Connected” in Cloudflare.

6.3 Configure SSH access in Cloudflare

In Cloudflare Zero Trust Dashboard > Networks > Tunnels
Click on your tunnel > Configure
Go to the Public Hostname tab
Click Add a public hostname
Configure:
- Subdomain: console
- Domain: yourdomain.org (your domain)
- Type: SSH
- URL: localhost:22
Save hostname

6.4 Create Access Application

Go to Access > Applications
Click Add an application
Select Self-hosted
Configure:
- Application name: SSH Console
- Session Duration: As needed (e.g. 24 hours)
- Application domain: console.yourdomain.org
Click Next
Create a Policy:
- Policy name: e.g. Allow Admin
- Action: Allow
- Include: Your email address or identity provider
Save

Step 7: Local SSH configuration

Install cloudflared on local Mac

brew install cloudflared

Create/extend SSH config

Add the following to ~/.ssh/config:

# Temporary Admin Node (can be removed after setup)
Host admin-node
    HostName 2a01:4f8:xxxx:xxxx::1
    User kubernetes-admin
    Port 22
    IdentityFile ~/.ssh/k8s-cluster_admin-node_key
    PreferredAuthentications publickey

# Control Node via Admin Node (temporary)
Host control-node-01
    HostName 10.0.0.2
    User kubernetes-admin
    Port 22
    IdentityFile ~/.ssh/k8s-cluster_control-node_key
    PreferredAuthentications publickey
    ProxyJump admin-node

# Worker Node via Admin Node (temporary)
Host worker-node-01
    HostName 10.0.0.3
    User kubernetes-admin
    Port 22
    IdentityFile ~/.ssh/k8s-cluster_worker-node_key
    PreferredAuthentications publickey
    ProxyJump admin-node

# Control Node via Cloudflare Tunnel (permanent)
Host console.yourdomain.org
    HostName console.yourdomain.org
    User kubernetes-admin
    IdentityFile ~/.ssh/k8s-cluster_control-node_key
    ProxyCommand cloudflared access ssh --hostname %h

Note

`cloudflared` must be installed and available in `$PATH` (`brew install cloudflared` on macOS, included in the Dev Container)

Test connection

# Via Cloudflare Tunnel
ssh console.yourdomain.org

On first connection, a browser window will open for Cloudflare Access authentication.

Step 8: Disable public IPs

After the Cloudflare Tunnel is working:

Hetzner Console > Servers > control-node-01
Networking > Public Network > Disable
Repeat for worker-node-01

The servers are now only reachable via the Cloudflare Tunnel (Control Node) or the internal network (Worker Node).

Step 9: Remove Admin Node

The temporary Admin Node is no longer needed:

Hetzner Console > Servers > admin-node
Delete > Confirm

Also remove the corresponding entries from ~/.ssh/config and update the ProxyJump entries:

# Worker Node via Cloudflare Tunnel (through Control Node)
Host worker-node-01
    HostName 10.0.0.3
    User kubernetes-admin
    Port 22
    IdentityFile ~/.ssh/k8s-cluster_worker-node_key
    PreferredAuthentications publickey
    ProxyJump console.yourdomain.org

Final SSH configuration

After completing all steps:

# Control Node via Cloudflare Tunnel
Host control-node-01
    HostName console.yourdomain.org
    User kubernetes-admin
    IdentityFile ~/.ssh/k8s-cluster_control-node_key
    ProxyCommand cloudflared access ssh --hostname %h

# Worker Node via Cloudflare Tunnel → Control Node → Internal network
Host worker-node-01
    HostName 10.0.0.3
    User kubernetes-admin
    Port 22
    IdentityFile ~/.ssh/k8s-cluster_worker-node_key
    PreferredAuthentications publickey
    ProxyJump control-node-01

References

6 - Reference

Configuration, outputs, templates, and project structure reference.

Use this section when you need exact variable names, output formats, templates, or operational reference material.

6.1 - Variables Reference

Every OpenTofu input variable with its type, default, and effect.

All configurable inputs for the OpenTofu infrastructure. Set these in terraform.tfvars.

Required Variables

Variable	Type	Description
`hcloud_token`	`string`	Hetzner Cloud API Token (sensitive)

Cluster Configuration

Variable	Type	Default	Description
`cluster_name`	`string`	`"k8s-cluster"`	Name prefix for all resources
`location`	`string`	`"fsn1"`	Hetzner Cloud location (`fsn1`, `nbg1`, `hel1`, `ash`, `hil`)

Network Configuration

Variable	Type	Default	Description
`network_ip_range`	`string`	`"10.0.0.0/8"`	IP range for the private network
`subnet_ip_range`	`string`	`"10.0.0.0/24"`	IP range for the subnet
`network_zone`	`string`	`"eu-central"`	Network zone (`eu-central`, `us-east`, `us-west`)
`enable_public_ipv6`	`bool`	`true`	Enable public IPv6 for replica control nodes and worker nodes. The master always has public IPv6 (required for cloudflared). Setting to `false` air-gaps replicas and workers.

Server Configuration

Variable	Type	Default	Description
`server_image`	`string`	`"debian-13"`	Server image to use
`master_control_node_type`	`string`	`"cx23"`	Server type for the master control node (runs cloudflared)
`control_node_types`	`list(object({type, count}))`	`[]`	Server types and counts for replica control nodes
`worker_node_types`	`list(object({type, count}))`	`[]`	Server types and counts for worker nodes

The cost-optimized x86 line is cx23, cx33, cx43, and cx53; the Arm equivalents are cax11 through cax41. See Cost Estimate for specifications, current prices, and the constraints on the Arm line.

Node type examples

# Master-only (default)
master_control_node_type = "cx23"
control_node_types = []
worker_node_types  = []

# 3 control nodes + 2 workers
control_node_types = [
  { type = "cx23", count = 2 },
]
worker_node_types = [
  { type = "cx23", count = 2 },
]

# Mixed worker types
worker_node_types = [
  { type = "cx23", count = 2 },
  { type = "cx33", count = 1 },
]

Authentication

Variable	Type	Default	Description
`root_password`	`string`	`"ChangeMe123!"`	Root password for emergency Web-Console access (sensitive)
`admin_user`	`string`	`"kubernetes-admin"`	Admin user name for SSH access
`keyboard_layout`	`string`	`"de"`	Keyboard layout for cloud-init

SSH Keys

If left empty, new keys will be auto-generated and stored in the OpenTofu state. When using custom keys, manage private keys yourself.

Variable	Type	Default	Description
`control_node_public_key`	`string`	`""`	Public SSH key for control nodes. Leave empty to auto-generate.
`worker_node_public_key`	`string`	`""`	Public SSH key for worker nodes. Leave empty to auto-generate.
`admin_node_public_key`	`string`	`""`	Public SSH key for admin node. Leave empty to auto-generate.
`ssh_key_prefix`	`string`	`""`	Prefix for SSH key filenames. Defaults to `cluster_name` if empty.

Cloudflare Configuration

Variable	Type	Default	Description
`cloudflare_tunnel_domain`	`string`	`""`	Domain for Cloudflare Tunnel SSH access (e.g., `console.example.org`)
`cloudflare_tunnel_token`	`string`	`""`	Cloudflare Tunnel token for automatic setup on master node (sensitive). Leave empty for manual setup.

Admin Node

Variable	Type	Default	Description
`enable_admin_node`	`bool`	`true`	Enable a temporary admin node with public IPv6 for initial SSH access. Disable after Cloudflare Tunnel is configured.
`admin_node_type`	`string`	`"cx23"`	Server type for admin node

NAT64/DNS64

Variable	Type	Default	Description
`enable_nat64`	`bool`	`true`	Enable NAT64/DNS64 for IPv4 reachability on IPv6-only nodes
`dns64_resolvers`	`list(string)`	`["2a01:4f8:c2c:123f::1", "2a01:4f9:c010:3f02::1", "2a00:1098:2b::1"]`	DNS64 resolver addresses (nat64.net: Nuremberg, Helsinki, Amsterdam)

Kubernetes

Variable	Type	Default	Description
`enable_k8s_prereqs`	`bool`	`true`	Install Kubernetes prerequisites (container runtime, kubeadm, kubelet, kubectl) via cloud-init
`kubernetes_version`	`string`	`"1.32"`	Kubernetes minor version for the pkgs.k8s.io apt source
`container_runtime`	`string`	`"containerd"`	Container runtime for Kubernetes nodes: `"containerd"` or `"cri-o"`

6.2 - Outputs Reference

Every OpenTofu output, what consumes it, and how to query it.

All outputs exposed by the OpenTofu configuration. These are consumed by the setup scripts and can be queried manually.

Network Information

Output	Description
`network_id`	ID of the private network
`network_name`	Name of the private network

Cluster Metadata

Output	Description
`cluster_name`	Name of the cluster
`admin_user`	Admin user name
`ssh_key_prefix`	Prefix used for SSH key filenames

Master Control Node

Output	Description
`master_control_node_id`	ID of the master control node
`master_control_node_name`	Name of the master control node
`master_control_node_private_ip`	Private IP of the master control node

All Control Nodes (master + replicas)

Output	Description
`control_node_count`	Total number of control nodes (master + replicas)
`control_node_ids`	IDs of all control nodes
`control_node_names`	Names of all control nodes
`control_node_private_ips`	Private IPs of all control nodes

Worker Nodes

Output	Description
`worker_node_count`	Number of worker nodes
`worker_node_ids`	IDs of worker nodes
`worker_node_names`	Names of worker nodes
`worker_node_private_ips`	Private IPs of worker nodes

Admin Node

Output	Description
`admin_node_id`	ID of the admin node (null if disabled)
`admin_node_name`	Name of the admin node (null if disabled)
`admin_node_ipv6`	Public IPv6 address of the admin node (null if disabled)
`admin_node_private_ip`	Private IP of the admin node (null if disabled)
`enable_admin_node`	Whether the admin node is enabled

SSH Keys

These outputs are sensitive when auto-generated keys are used.

Output	Sensitive	Description
`control_node_ssh_private_key`	Yes	Private SSH key for control nodes (only if auto-generated)
`control_node_ssh_public_key`	No	Public SSH key for control nodes
`worker_node_ssh_private_key`	Yes	Private SSH key for worker nodes (only if auto-generated)
`worker_node_ssh_public_key`	No	Public SSH key for worker nodes
`admin_node_ssh_private_key`	Yes	Private SSH key for admin node (only if auto-generated)
`admin_node_ssh_public_key`	No	Public SSH key for admin node
`using_custom_keys`	No	Map showing which node roles use custom keys

SSH Config

Output	Description
`ssh_config_snippet`	SSH config snippet for `~/.ssh/config`. Generates Host entries for admin-node (ProxyJump), cloudflare tunnel (ProxyCommand), and all nodes. All entries include `IdentitiesOnly yes`.

Feature Flags

Output	Description
`cloudflare_tunnel_domain`	Configured Cloudflare Tunnel domain
`cloudflare_tunnel_configured`	Whether `cloudflare_tunnel_token` was set, i.e. whether cloud-init installed the tunnel automatically
`nat64_enabled`	Whether NAT64/DNS64 is enabled
`k8s_prereqs_enabled`	Whether Kubernetes prerequisites are installed via cloud-init

Output	Description
`next_steps`	Instructions banner displayed after `tofu apply`

Querying outputs

# List all outputs
tofu output

# Get a specific output
tofu output master_control_node_private_ip

# Get a sensitive output (raw)
tofu output -raw control_node_ssh_private_key

# Export SSH key to file
tofu output -raw control_node_ssh_private_key > ~/.ssh/k8s-cluster_control-node_key
chmod 600 ~/.ssh/k8s-cluster_control-node_key

6.3 - Cloud-Init Templates

The three cloud-init templates, their template variables, and their conditional sections.

Cloud-init templates are located in cloud-init/ and rendered by OpenTofu via templatefile() in main.tf. They configure each server on first boot.

admin-node.yaml.tpl

Purpose: Minimal jump host for initial SSH access.

Used by: hcloud_server.admin_node

Template variables:

Variable	Source
`ssh_public_key`	`local.admin_node_public_key`
`root_password`	`var.root_password`
`admin_user`	`var.admin_user`
`keyboard_layout`	`var.keyboard_layout`

What it configures:

Admin user with sudo NOPASSWD
SSH hardening with AllowTcpForwarding yes (needed for ProxyJump)
fail2ban for SSH protection
UFW allowing public SSH (port 22 from anywhere)

control-node.yaml.tpl

Purpose: Kubernetes control plane node with optional Cloudflare Tunnel.

Used by: hcloud_server.master_control_node, hcloud_server.control_node_replica

Template variables:

Variable	Source
`ssh_public_key`	`local.control_node_public_key`
`root_password`	`var.root_password`
`admin_user`	`var.admin_user`
`keyboard_layout`	`var.keyboard_layout`
`is_master`	`true` for master, `false` for replicas
`enable_nat64`	`var.enable_nat64`
`dns64_resolvers`	`var.dns64_resolvers`
`enable_k8s_prereqs`	`var.enable_k8s_prereqs`
`kubernetes_version`	`var.kubernetes_version`
`container_runtime`	`var.container_runtime`
`cloudflare_tunnel_token`	`var.cloudflare_tunnel_token` (master), `""` (replicas)

What it configures:

Admin user with sudo NOPASSWD and root password for emergency console access
SSH hardening (key-only auth, AllowTcpForwarding yes)
fail2ban for SSH protection
UFW: SSH from internal network (10.0.0.0/8) + localhost (for Cloudflare Tunnel)

Conditional sections:

is_master = true: Installs cloudflared, allows SSH from localhost, creates /etc/cloudflared/config.yml with edge-ip-version: "6". When cloudflare_tunnel_token is set, runs cloudflared service install <token> to auto-configure the tunnel as a systemd service.
is_master = false: Skips cloudflared installation (token always empty for replicas)
enable_nat64 = true: Configures DNS64 resolvers in systemd-resolved, adds NAT64 route (64:ff9b::/96), creates networkd-dispatcher persistence script
enable_k8s_prereqs = true: Installs the selected container runtime (containerd with SystemdCgroup and sandbox image fix, or cri-o from OBS repo with kubelet socket configuration), kubeadm, kubelet, kubectl, loads kernel modules (overlay, br_netfilter), sets sysctl params, disables swap, opens kubelet (10250) + etcd (2379-2380) ports

worker-node.yaml.tpl

Purpose: Kubernetes worker node with restricted network access.

Used by: hcloud_server.worker_node

Template variables:

Variable	Source
`ssh_public_key`	`local.worker_node_public_key`
`root_password`	`var.root_password`
`admin_user`	`var.admin_user`
`keyboard_layout`	`var.keyboard_layout`
`enable_nat64`	`var.enable_nat64`
`dns64_resolvers`	`var.dns64_resolvers`
`enable_k8s_prereqs`	`var.enable_k8s_prereqs`
`kubernetes_version`	`var.kubernetes_version`
`container_runtime`	`var.container_runtime`

What it configures:

Admin user with sudo NOPASSWD and root password for emergency console access
SSH hardening with AllowTcpForwarding no (workers cannot be used as jump hosts)
fail2ban for SSH protection
UFW: deny all by default, SSH from internal network only, outbound limited to DNS + HTTP/S

Conditional sections:

enable_nat64 = true: Uses DNS64 resolvers instead of Hetzner DNS, adds NAT64 prefix UFW rule
enable_k8s_prereqs = true: Same as control node prerequisites (runtime selected by container_runtime), but without etcd ports

Template rendering

Templates are rendered in main.tf via templatefile():

user_data = templatefile("${path.module}/cloud-init/control-node.yaml.tpl", {
  ssh_public_key         = local.control_node_public_key
  root_password          = var.root_password
  admin_user             = var.admin_user
  keyboard_layout        = var.keyboard_layout
  is_master              = true
  enable_nat64           = var.enable_nat64
  dns64_resolvers        = var.dns64_resolvers
  enable_k8s_prereqs     = var.enable_k8s_prereqs
  kubernetes_version     = var.kubernetes_version
  container_runtime      = var.container_runtime
  cloudflare_tunnel_token = var.cloudflare_tunnel_token
})

Note

The `lifecycle { ignore_changes = [user_data] }` block on all servers prevents re-provisioning when template variables change. Cloud-init only runs on first boot. Use [Ansible playbooks](/kubeclaw/docs/reference/playbooks/) for configuration changes on running nodes.

6.4 - Ansible Playbooks Reference

Purpose, variables, and task list for each Ansible playbook.

All playbooks are in ansible/playbooks/ and target the k8s_cluster host group by default.

update-system.yml

Purpose: Run apt update && apt upgrade on all nodes.

Usage:

cd ansible
ansible-playbook playbooks/update-system.yml
ansible-playbook playbooks/update-system.yml --limit control_nodes
ansible-playbook playbooks/update-system.yml -e "reboot_after_update=true"

Variables:

Variable	Default	Description
`reboot_after_update`	`false`	Reboot after update if the system requires it
`reboot_timeout`	`300`	Timeout (seconds) to wait for reboot

Tasks:

Update apt cache
Upgrade all packages (dist-upgrade with autoremove)
Check if reboot is required (/var/run/reboot-required)
Reboot if required and enabled
Wait for system to come back online

security-hardening.yml

Purpose: Apply security measures on all nodes.

Usage:

cd ansible
ansible-playbook playbooks/security-hardening.yml

Variables:

Variable	Default	Description
`enable_unattended_upgrades`	`true`	Install and configure unattended-upgrades
`configure_fail2ban`	`true`	Verify fail2ban is running

Tasks:

Install and configure unattended-upgrades (security updates only)
Enable automatic update checks (daily)
Verify fail2ban is running and monitoring SSH
Set secure permissions on /etc/shadow
Disable core dumps
Apply sysctl hardening (reverse path filtering, source route rejection, ASLR)

configure-nat64.yml

Purpose: Configure DNS64 resolvers and NAT64 routing on already-running nodes. Cloud-init only runs at first boot – use this playbook for existing nodes or to reconfigure.

Usage:

cd ansible
ansible-playbook playbooks/configure-nat64.yml
ansible-playbook playbooks/configure-nat64.yml --limit control_nodes
ansible-playbook playbooks/configure-nat64.yml -e '{"dns64_resolvers":["2a01:4f8:c2c:123f::1"]}'

Variables:

Variable	Default	Description
`dns64_resolvers`	`["2a01:4f8:c2c:123f::1", "2a01:4f9:c010:3f02::1", "2a00:1098:2b::1"]`	DNS64 resolver addresses
`nat64_prefix`	`"64:ff9b::/96"`	NAT64 prefix

Tasks:

Configure DNS64 resolvers in systemd-resolved
Add NAT64 route via default IPv6 gateway
Create networkd-dispatcher script for persistent route
Update UFW rules on worker nodes (remove Hetzner DNS, add DNS64 + NAT64 rules)
Verify DNS64 resolution and NAT64 connectivity

prepare-k8s-nodes.yml

Purpose: Install container runtime (containerd or CRI-O), kubeadm, kubelet, and kubectl on already-running nodes. Cloud-init only runs at first boot – use this playbook for existing nodes.

Usage:

cd ansible
ansible-playbook playbooks/prepare-k8s-nodes.yml
ansible-playbook playbooks/prepare-k8s-nodes.yml --limit control_nodes
ansible-playbook playbooks/prepare-k8s-nodes.yml -e "kubernetes_version=1.32"
ansible-playbook playbooks/prepare-k8s-nodes.yml -e "container_runtime=cri-o"

Variables:

Variable	Default	Description
`kubernetes_version`	`"1.32"`	Kubernetes minor version for apt repo
`container_runtime`	`"containerd"`	Container runtime: `"containerd"` or `"cri-o"`

Tasks:

Load kernel modules (overlay, br_netfilter)
Set sysctl parameters (bridge-nf-call-iptables, ip_forward)
Disable swap
Install and configure container runtime (containerd with SystemdCgroup, or CRI-O from OBS repo)
Add Kubernetes apt repository
Install kubelet, kubeadm, kubectl (held at current version)
Open UFW ports on control nodes (kubelet 10250, etcd 2379-2380)
Verify kubeadm version and container runtime status

Ansible configuration

The ansible.cfg file configures:

Remote user: kubernetes-admin
Host key checking: disabled (nodes are ephemeral)
Privilege escalation: sudo NOPASSWD
SSH args: ForwardAgent, ControlMaster, ControlPersist for fast connections
Pipelining: enabled for performance

6.5 - OS Images

Hetzner Cloud OS images available for cluster nodes, and why the default is Debian 13.

Hetzner Cloud offers the following Debian/Ubuntu images:

Image	Name	Recommendation
`debian-13`	Debian 13 Trixie	Recommended for K8s
`debian-12`	Debian 12 Bookworm	Stable, well-proven
`ubuntu-24.04`	Ubuntu 24.04 LTS	Good for K8s
`ubuntu-22.04`	Ubuntu 22.04 LTS	Well-proven

Why Debian 13?

Stability: long support cycles
Compatibility: the pkgs.k8s.io apt repository used for kubeadm, kubelet, and kubectl ships Debian packages
Small footprint: leaner than Ubuntu, but not as small as Alpine
No glibc/musl issues: unlike Alpine, no compatibility problems

Hetzner Cloud doesn’t provide dedicated “slim” or “minimal” variants. The standard images are fairly compact already.

6.6 - Project Structure

Repository layout and the purpose of each top-level file and directory.

kubeclaw/
├── main.tf                          # Infrastructure (providers, network, firewalls, servers)
├── variables.tf                     # All configurable inputs
├── outputs.tf                       # IPs, SSH keys, ssh_config_snippet, next-steps banner
├── terraform.tfvars.example         # Example configuration
├── hugo.yaml                        # Hugo + Docsy configuration
├── package.json                     # Docsy asset dependencies
├── themes/docsy/                    # Pinned Docsy theme submodule
├── aibox.toml                       # Dev Container source of truth (aibox apply)
├── aibox.lock                       # Pinned aibox toolchain versions
├── CLAUDE.md                        # Claude Code project instructions
├── README.md                        # Project overview (concise)
├── LICENSE                          # MIT License
├── .gitignore
├── .devcontainer/
│   ├── devcontainer.json            # Generated by aibox; do not edit directly
│   ├── Dockerfile                   # Generated by aibox; do not edit directly
│   ├── docker-compose.yml           # Generated by aibox; do not edit directly
│   ├── Dockerfile.local             # Project layer: installs cloudflared
│   └── docker-compose.override.yml  # Project override: writable SSH mount
├── cloud-init/
│   ├── admin-node.yaml.tpl         # Admin node (temporary jump host with public IPv6)
│   ├── control-node.yaml.tpl       # Control node (cloudflared on master, UFW, fail2ban, k8s prereqs)
│   └── worker-node.yaml.tpl        # Worker node (isolated, outbound DNS/HTTP/S only, k8s prereqs)
├── scripts/
│   ├── setup-ssh.sh                # Export SSH keys from tofu state, generate ~/.ssh/config
│   ├── ssh-agent-setup.sh          # Fix SSH permissions, start ssh-agent, load keys
│   ├── generate-ansible-inventory.sh  # Build ansible/inventory.ini from tofu state
│   ├── build-docs.sh              # Build Hugo site locally
│   ├── serve-docs.sh              # Preview Hugo site locally
│   └── deploy-docs.sh             # Build and deploy Hugo to gh-pages
├── ansible/
│   ├── ansible.cfg                  # Ansible defaults (user, pipelining, SSH args)
│   ├── inventory.ini                # Auto-generated inventory (do not hand-edit)
│   └── playbooks/
│       ├── update-system.yml        # System updates with optional reboot
│       ├── security-hardening.yml   # Unattended upgrades, fail2ban, sysctl hardening
│       ├── configure-nat64.yml      # NAT64/DNS64 for IPv4 reachability on running nodes
│       └── prepare-k8s-nodes.yml   # Kubernetes prerequisites (container runtime, kubeadm) on running nodes
├── assets/                         # Project brand layer for the Hugo site
│   ├── icons/logo.svg              # projectious mark used in the navbar
│   └── scss/                       # Brand tokens and Docsy surface styles
├── layouts/                        # Project template overrides (favicons, head hooks)
├── static/favicons/                # Brand favicon set
├── content/                        # Hugo documentation source
│   ├── _index.md                   # Branded landing page
│   └── docs/                       # Documentation section
│       ├── _index.md               # Documentation overview
│       ├── quick-start.md          # Combined prerequisites + deployment steps
│       ├── introduction/           # Concepts: architecture, security, DNS/NAT64
│       ├── guide/                  # Ordered deployment path (dev container -> OpenClaw)
│       ├── how-to/                 # Focused procedures, incl. the manual setup alternative
│       ├── reference/              # Variables, outputs, templates, playbooks, structure
│       ├── operations/             # Day-two: scaling, rotation, upgrades, troubleshooting
│       ├── roadmap/                # Planned work
│       └── contributing/           # Development setup and code of conduct
└── public/                         # Generated site output (gitignored)

Key Files

Infrastructure (root level)

File	Purpose
`main.tf`	Core infrastructure: providers, SSH keys, network/subnet, firewalls, servers, cloud-init rendering
`variables.tf`	All configurable inputs with types, defaults, and descriptions
`outputs.tf`	Exposes IPs, SSH keys (sensitive), ssh_config_snippet, and next-steps banner
`terraform.tfvars.example`	Reference configuration (actual `.tfvars` is gitignored)

Cloud-Init Templates

File	Purpose
`cloud-init/admin-node.yaml.tpl`	Minimal jump host: admin user, SSH hardening, fail2ban, public SSH
`cloud-init/control-node.yaml.tpl`	Control plane: cloudflared (master only), UFW, fail2ban, optional NAT64 + K8s prereqs
`cloud-init/worker-node.yaml.tpl`	Worker: restrictive UFW, no TCP forwarding, optional NAT64 + K8s prereqs

Scripts

File	Purpose
`scripts/setup-ssh.sh`	Export SSH keys, generate `~/.ssh/config` with backup
`scripts/ssh-agent-setup.sh`	Fix SSH permissions, start ssh-agent, load keys
`scripts/generate-ansible-inventory.sh`	Build Ansible inventory from OpenTofu state
`scripts/build-docs.sh`	Build the Hugo site locally
`scripts/serve-docs.sh`	Preview the site locally
`scripts/deploy-docs.sh`	Build Hugo and push to `gh-pages`

Dev Container

The aibox configuration in aibox.toml is the source of truth for generated Dev Container files. Run aibox apply after changing it; do not hand-edit .devcontainer/Dockerfile, docker-compose.yml, or devcontainer.json. Dockerfile.local and docker-compose.override.yml are project-owned extensions and are safe to edit.

Ansible

File	Purpose
`ansible/ansible.cfg`	Defaults: remote user, host key checking, privilege escalation
`ansible/inventory.ini`	Auto-generated inventory with `control_nodes`, `worker_nodes`, `k8s_cluster` groups

Documentation site

File	Purpose
`hugo.yaml`	Hugo + Docsy configuration, menus, version list, and module mounts
`assets/scss/_variables_project.scss`	projectious brand tokens mapped onto Bootstrap/Docsy variables
`assets/scss/_styles_project.scss`	Brand surface styles: type scale, code theme, dark mode, chrome
`assets/icons/logo.svg`	The projectious mark inlined into the navbar
`layouts/partials/favicons.html`	Favicon set override (Docsy’s default references assets this project does not ship)
`layouts/partials/hooks/head-end.html`	Brand web-font loading
`static/favicons/`	Brand favicon and touch-icon assets

Module mounts

`hugo.yaml` declares `module.mounts` for the vendored Bootstrap and Font Awesome assets. Declaring any mount for a component replaces Hugo's default mount for that component, so the config also restores `assets` and `static` explicitly. Removing those two entries silently disables every project-level brand asset.

6.7 - Cost Estimate

Indicative monthly Hetzner Cloud costs for common cluster sizes.

All figures are list prices in EUR excluding VAT, current as of the Hetzner price adjustment of 15 June 2026. Hetzner bills hourly; the monthly figure is the cap you will not exceed. Always check Hetzner Cloud pricing before committing – prices changed several times during 2026.

Server types

KubeClaw defaults to the cost-optimized CX line (shared vCPU, x86). The CAX line offers the same resources on Ampere Arm cores, but is only available in the German and Finnish locations (fsn1, nbg1, hel1) and requires arm64 container images throughout the cluster.

Type	vCPU	RAM	NVMe	Monthly (excl. IPv4)
`cx23`	2	4 GB	40 GB	€5.49
`cx33`	4	8 GB	80 GB	€8.49
`cx43`	8	16 GB	160 GB	€15.99
`cx53`	16	32 GB	320 GB	€29.49
`cax11` (Arm)	2	4 GB	40 GB	€5.99
`cax21` (Arm)	4	8 GB	80 GB	€10.49
`cax31` (Arm)	8	16 GB	160 GB	€20.99
`cax41` (Arm)	16	32 GB	320 GB	€40.99

The CX22 generation is retired

Earlier revisions of this project defaulted to `cx22`. That generation (`cx22`/`cx32`/`cx42`/`cx52`) is no longer offered for new orders -- the current cost-optimized line is `cx23`/`cx33`/`cx43`/`cx53`. Existing servers keep their original pricing until they are rescaled or recreated.

The higher-performance CPX (dedicated AMD share) and CCX (fully dedicated) lines are considerably more expensive – cpx22 is €19.49/month and ccx13 is €42.99/month – and are rarely worth it for a learning or small-production cluster.

Additional costs

Item	Price
Primary IPv4 address	€0.50 per server per month
Block storage volume	~€0.0572 per GB per month
Private network	Free
Cloudflare Tunnel + Access	Free (up to 50 users)

IPv6-only is a real saving

KubeClaw provisions every node without a public IPv4 address, which avoids the €0.50 per server per month primary-IPv4 charge. On a five-node cluster that is €30/year, and it is the reason [NAT64/DNS64](/kubeclaw/docs/introduction/dns-and-nat64/) exists in this project.

Example: 2-node cluster

Component	Specification	Monthly Cost
Master control node (`cx23`, IPv6-only)	2 vCPU / 4 GB RAM	€5.49
Worker node (`cx23`, IPv6-only)	2 vCPU / 4 GB RAM	€5.49
Block volumes (10 GB each)	Hetzner CSI, 20 GB total	~€1.14
Private network	–	Free
Cloudflare Tunnel + Access	Up to 50 users	Free
Total		~€12.12/month

Scaling costs

Server cost only; add block volumes for any workload that needs persistent storage.

Configuration	Nodes	Server cost
Master-only (dev/learning)	1	€5.49
Master + 1 worker	2	€10.98
3 control + 2 workers (HA)	5	€27.45

Notes

The admin node (cx23, €5.49/month) is temporary. Set enable_admin_node = false once the Cloudflare Tunnel works, and the charge stops – see Quick Start.
Mixed server types are supported, so workers can be sized independently of the control plane. See Scale Up/Down.
Snapshots and backups are billed separately and are not included above.

7 - Operations

Keep the cluster secure, healthy, and maintainable after deployment.

Day-two procedures for scaling, maintenance, troubleshooting, and security.

7.1 - Scale Up/Down

Add or remove control and worker nodes at the infrastructure and Kubernetes levels.

This guide covers scaling your cluster at both the infrastructure and Kubernetes levels.

Scale Infrastructure

Add worker nodes

Edit terraform.tfvars – update worker_node_types:

worker_node_types = [
  { type = "cx23", count = 2 },
  { type = "cx33", count = 1 },  # 3 workers total, mixed types
]

Apply changes:
```
tofu apply
```

Regenerate Ansible inventory:

./scripts/generate-ansible-inventory.sh

Run Ansible playbooks on the new nodes – see Server Management (Ansible) for the full workflow.

Add replica control nodes

Same pattern using control_node_types:

control_node_types = [
  { type = "cx23", count = 2 },  # 2 replicas → 3 total control nodes
]

Then tofu apply, regenerate inventory, and run Ansible playbooks.

Remove nodes

Drain and remove from Kubernetes first (see Remove nodes from Kubernetes below)
Update terraform.tfvars to reduce node counts
Run tofu apply

Scale Kubernetes

Join new worker nodes

After provisioning and running Ansible playbooks on new nodes, join them to the cluster. Follow Step 2 of the Kubernetes guide to run the kubeadm join command.

Remove nodes from Kubernetes

Before removing infrastructure, drain and delete the node from Kubernetes:

kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data
kubectl delete node <node-name>

Then update terraform.tfvars and run tofu apply to remove the infrastructure.

Examples

Master-only (no replicas, no workers)

# terraform.tfvars
master_control_node_type = "cx23"
control_node_types = []
worker_node_types  = []

Add 2 workers

# terraform.tfvars
worker_node_types = [
  { type = "cx23", count = 2 },
]

Mixed server types

# terraform.tfvars
control_node_types = [
  { type = "cx23", count = 2 },
]
worker_node_types = [
  { type = "cx23", count = 2 },
  { type = "cx33", count = 1 },
]

7.2 - SSH Key Rotation

Rotate auto-generated or custom SSH keys without losing access.

When to rotate

Periodically (e.g. annually)
If compromise is suspected
When personnel changes occur

With auto-generated keys

# 1. Mark old key resources for recreation
tofu taint 'tls_private_key.control_node[0]'
tofu taint 'tls_private_key.worker_node[0]'

# 2. Generate new keys and update servers
tofu apply

# 3. Export new keys
./scripts/setup-ssh.sh

Warning

During rotation SSH access may be briefly interrupted. Keep Hetzner web console root access available as a fallback.

With custom keys

# 1. Create new keys
ssh-keygen -t ed25519 -f ~/.ssh/k8s-control-new -C "control-node-new"

# 2. Add the new public key to the server (before removing the old one)
ssh control-node
echo "ssh-ed25519 AAAA... control-node-new" >> ~/.ssh/authorized_keys

# 3. Test the new key
ssh -i ~/.ssh/k8s-control-new kubernetes-admin@<node-ip>

# 4. Remove the old key from authorized_keys
ssh -i ~/.ssh/k8s-control-new control-node
# Edit ~/.ssh/authorized_keys and remove the old key line

# 5. Update terraform.tfvars with the new public key
# control_node_public_key = "ssh-ed25519 AAAA... (new key)"

# 6. Sync OpenTofu state
tofu apply

Repeat for worker node keys if applicable.

7.3 - Kubernetes Maintenance

Upgrade the control plane and workers, back up volumes, and inspect cluster state.

Operational procedures for upgrading and managing the Kubernetes cluster. For initial cluster setup, see the Kubernetes (kubeadm) guide.

Upgrade Kubernetes

Kubernetes upgrades follow a strict order: control plane first, then workers. This is the standard CKA upgrade workflow.

Upgrade control plane

# 1. Unhold packages
sudo apt-mark unhold kubeadm

# 2. Upgrade kubeadm
sudo apt-get update && sudo apt-get install -y kubeadm=1.33.*-*

# 3. Check available upgrade
sudo kubeadm upgrade plan

# 4. Apply the upgrade
sudo kubeadm upgrade apply v1.33.0

# 5. Drain the control node (if running workloads)
kubectl drain $(hostname) --ignore-daemonsets --delete-emptydir-data

# 6. Upgrade kubelet and kubectl
sudo apt-mark unhold kubelet kubectl
sudo apt-get install -y kubelet=1.33.*-* kubectl=1.33.*-*
sudo apt-mark hold kubelet kubeadm kubectl
sudo systemctl daemon-reload
sudo systemctl restart kubelet

# 7. Uncordon the node
kubectl uncordon $(hostname)

Upgrade worker nodes

On each worker node:

# 1. From the control plane: drain the worker
kubectl drain <worker-name> --ignore-daemonsets --delete-emptydir-data

# 2. On the worker: upgrade packages
sudo apt-mark unhold kubeadm kubelet kubectl
sudo apt-get update && sudo apt-get install -y kubeadm=1.33.*-* kubelet=1.33.*-* kubectl=1.33.*-*
sudo apt-mark hold kubeadm kubelet kubectl

# 3. Upgrade node config
sudo kubeadm upgrade node

# 4. Restart kubelet
sudo systemctl daemon-reload
sudo systemctl restart kubelet

# 5. From the control plane: uncordon the worker
kubectl uncordon <worker-name>

Backup PVC data

# Create snapshot via Hetzner Console or API
# Hetzner Console → Volumes → Select volume → Create Snapshot

Useful kubectl commands

# Check nodes (dual-stack IPs visible)
kubectl get nodes -o wide

# Check all pods across namespaces
kubectl get pods -A

# Check pod IPs (should show both IPv4 and IPv6)
kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.name}: {.status.podIPs}{"\n"}{end}'

# Check storage
kubectl get pvc -A
kubectl get pv

# Check CSI driver
kubectl get pods -n kube-system | grep hcloud

# Check Cilium status
kubectl -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status

# Check network policies
kubectl get ciliumnetworkpolicies -A

# Check FQDN DNS cache (verify DNS64 synthesized addresses are cached)
kubectl exec -n kube-system -it \
  $(kubectl get pods -n kube-system -l k8s-app=cilium -o name | head -1) \
  -- cilium fqdn cache list

# Logs for cloudflared (if running as K8s workload)
kubectl logs -n system-unrestricted -l app=cloudflared

# Restart a workload
kubectl rollout restart deployment/my-app -n apps-restricted

7.4 - Security

What the setup protects against, what to monitor, and the exact SSH hardening applied.

Design intent, not a security guarantee

This page describes what the design *intends* to protect against. KubeClaw is a [prototype](/kubeclaw/docs/project-status/) and has not been adversarially tested or audited -- assume gaps exist. Treat the table below as a statement of intent, not of assurance.

Security Summary

Layer	Protection
Network (Hetzner)	Firewall blocks all inbound; IPv6-only, no public IPv4
Network (K8s)	Cilium egress whitelist per namespace/app (FQDN-based)
Access	Cloudflare Tunnel (outbound-only connection, no open ports)
Authentication	Cloudflare Access policies + SSH key-only auth
Container	Non-root user, dropped capabilities, resource limits
SSH	Key-only auth, fail2ban, no TCP forwarding on workers
Storage	Isolated PVCs per workload

What this setup protects against

Direct server attacks – no public IPs, no open inbound ports
Unauthorized access – Cloudflare Access + SSH key-only auth
Data exfiltration – FQDN-based egress whitelist per application
Lateral movement – namespace isolation, per-pod network policies
Resource abuse – container resource limits

What to monitor

API key and token compromise – rotate regularly
Cloudflare Tunnel health – monitor via Zero Trust dashboard
Node resource utilization – watch for memory pressure on small instances

Security notes

Passwords in cloud-init are visible in cloud-init logs. Change them after first login.
SSH keys should be different for each server role.
Root password is only intended for emergency access via Hetzner Web Console.
Terraform state contains sensitive data (private keys when auto-generated). Protect state files.

UFW rules for HTTP/HTTPS on worker nodes can be removed after initial setup:

sudo ufw delete allow out to any port 80 proto tcp
sudo ufw delete allow out to any port 443 proto tcp

OpenClaw-specific security

When running OpenClaw:

Concern	Mitigation
Anthropic API key compromise	Rotate regularly, monitor usage
Telegram bot token leak	Monitor bot activity
Claude providing incorrect information	Human review of responses

SSH hardening details

All nodes are configured with:

PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
MaxAuthTries 3
X11Forwarding no
AllowAgentForwarding no
AllowUsers kubernetes-admin
ClientAliveInterval 300
ClientAliveCountMax 2

Control nodes additionally allow AllowTcpForwarding yes (needed for ProxyJump and tunnel). Worker nodes set AllowTcpForwarding no.

7.5 - Password Management

Which credentials exist, where they live, and how to keep terraform.tfvars safe.

Which credentials exist?

Credential	Purpose	Storage
Hetzner API Token	Create infrastructure	`terraform.tfvars`
Root password	Emergency web console	`terraform.tfvars`
SSH private keys	Server access	`~/.ssh/` or password manager
Cloudflare Tunnel token	Tunnel auth	Cloudflare Dashboard

Recommended Dashlane structure

📁 K8s Cluster
├── 🔐 Hetzner API Token
│   └── Token: xxx
├── 🔐 Root Password
│   └── Password: xxx
├── 📝 SSH Keys (Secure Note)
│   ├── Control Node Private Key: ...
│   ├── Control Node Public Key: ...
│   ├── Worker Node Private Key: ...
│   └── Worker Node Public Key: ...
└── 🔐 Cloudflare Tunnel Token
    └── Token: xxx

Securing terraform.tfvars

terraform.tfvars contains sensitive data. Options:

Do not commit: exclude via .gitignore (default)
Encrypt: with git-crypt or sops

Use environment variables instead of tfvars:

export TF_VAR_hcloud_token="xxx"
export TF_VAR_root_password="xxx"

7.6 - Troubleshooting

Fixes for common infrastructure, tunnel, Ansible, and Kubernetes failures.

Infrastructure Issues

cloudflared won’t start (IPv6-only)

Check /etc/cloudflared/config.yml:

edge-ip-version: "6"

Important

The value must be a string in quotes (`"6"`, not `6`). Cloudflared defaults to IPv4 connections to Cloudflare's edge servers.

SSH key rotation failed

Connect via Hetzner web console (root password)

Add the new key manually:

echo "ssh-ed25519 AAAA..." >> /home/kubernetes-admin/.ssh/authorized_keys

Ansible cannot connect

Check:

Is cloudflared installed locally?
Is the tunnel running? (cloudflared tunnel list)
Is the inventory correct? (./scripts/generate-ansible-inventory.sh)
Is the ssh-agent running with keys loaded? (ssh-add -l)

State lost / keys gone

With auto-generated keys:

Connect via web console (root)
Create new keys
Add them to authorized_keys

Import servers into new state:

tofu import hcloud_server.master_control_node <server-id>

SSH connection via tunnel fails

Checklist:

Is cloudflared installed locally? (which cloudflared)
Is the ProxyCommand path correct?
Is the Cloudflare Access Application configured?
Is the tunnel shown as “Connected” in Cloudflare?

Hetzner Web Console does not work

Use the “Send Clipboard” button above the console
Use a simple password without special characters for the initial login
Alternative: Create a temporary Admin Node with a public IP

Cloud-Init password does not work

Cause: The old chpasswd.list syntax is deprecated.

Solution: Use the new syntax:

users:
  - name: root
    plain_text_passwd: 'your-password'
    lock_passwd: false

Kubernetes Issues

Worker nodes not joining

# On the worker node, check kubelet logs
journalctl -xeu kubelet

# Common issues:
# - Swap not disabled: swapoff -a
# - containerd not running: systemctl status containerd
# - Port 6443 not reachable: curl -k https://10.0.0.2:6443
# - Token expired (24h default): kubeadm token create --print-join-command

Cilium pods not ready

# Check Cilium status
kubectl -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg status

# Check Cilium pod logs
kubectl logs -n kube-system -l k8s-app=cilium

# Check Cilium pods
kubectl get pods -n kube-system -l k8s-app=cilium

Network policy blocking traffic unexpectedly

# Check which policies apply
kubectl get ciliumnetworkpolicies -n apps-restricted

# Check if FQDN rules are resolving
kubectl exec -n kube-system -it \
  $(kubectl get pods -n kube-system -l k8s-app=cilium -o name | head -1) \
  -c cilium-agent -- cilium-dbg fqdn cache list

CSI volume not attaching

kubectl describe pvc -n apps-restricted
kubectl get events -n apps-restricted --sort-by='.lastTimestamp'

Pod not starting (general)

kubectl describe pod -n <namespace> <pod-name>
kubectl logs -n <namespace> <pod-name> --previous

OpenClaw Pod not starting

kubectl describe pod -n apps-restricted -l app=openclaw
kubectl logs -n apps-restricted -l app=openclaw --previous

Cloudflare Tunnel not connecting (K8s workload)

kubectl logs -n system-unrestricted -l app=cloudflared

8 - Roadmap

Planned improvements and current infrastructure milestones.

Track the next improvements to KubeClaw and its IPv6-first Kubernetes platform.

8.1 - Cilium Dual-Stack Pod Network

Planned work on the Cilium dual-stack pod network.

Implemented

This feature has been implemented. The [Kubernetes guide](/kubeclaw/docs/guide/kubernetes/) now uses dual-stack `kubeadm init` and Cilium with `ipv6.enabled=true` as the standard configuration.

Summary

The cluster uses dual-stack networking (IPv4 + IPv6) in the Cilium pod network. Every pod gets both an IPv4 address (from 10.244.0.0/16) for internal cluster communication and an IPv6 address (from fd00:10:244::/48) for external connectivity via DNS64/NAT64.

This eliminates the need for hostNetwork: true on any pod, which means:

Cilium FQDN egress policies apply to all pods (CoreDNS, CSI controller, OpenClaw)
No port conflicts from hostNetwork bindings
Full pod network isolation is maintained by Cilium

How It Works

kubeadm init with dual-stack CIDRs: --pod-network-cidr=10.244.0.0/16,fd00:10:244::/48 --service-cidr=10.96.0.0/12,fd00:10:96::/108
Cilium with ipv4.enabled=true, ipv6.enabled=true, enableIPv6Masquerade=true
VXLAN tunnel runs over IPv4 underlay (proven, stable – nodes communicate via 10.0.0.0/24)
CoreDNS forwards to DNS64 resolvers (2001:67c:2b0::4) which synthesize AAAA records for IPv4-only domains
Pods route to 64:ff9b::/96 NAT64 addresses via their IPv6 address; Cilium masquerades to the node’s public IPv6
Cilium DNS proxy intercepts DNS64-synthesized AAAA responses and maps them to FQDNs for policy enforcement

Research Findings

The following questions were investigated before implementation:

Question	Answer
Does `ipv6.enabled=true` require kubeadm dual-stack CIDRs?	Yes – kubeadm must be initialized with both IPv4 and IPv6 CIDRs. CIDRs cannot be changed after init.
Can Cilium manage IPv6 IPAM independently?	Yes (cluster-pool mode), but Kubernetes IPAM with kubeadm-allocated CIDRs is simpler.
Does VXLAN work with IPv6 pod addresses?	Yes – VXLAN tunnel runs over IPv4 underlay (auto mode), encapsulating both IPv4 and IPv6 pod traffic.
Do FQDN egress rules work with DNS64?	Yes – Cilium’s DNS proxy intercepts all DNS responses (A and AAAA). DNS64-synthesized AAAA records are just AAAA records from Cilium’s perspective.
Can pods route to `64:ff9b::/96`?	Yes – via `enableIPv6Masquerade=true` (default). Pod IPv6 traffic is masqueraded to the node’s public IPv6.
Does CoreDNS still need `hostNetwork`?	No – with dual-stack, CoreDNS has an IPv6 pod address and can reach DNS64 resolvers via masquerade.

References

Cilium IPv6 tunnel support (PR #40324) – merged into v1.19.0
Cilium Cluster-Pool IPAM
Kubernetes dual-stack docs
kubeadm dual-stack support
CoreDNS dns64 plugin
DNS and NAT64 on KubeClaw

9 - Contributing

Help improve KubeClaw and its documentation.

Read the contribution guidelines, development setup, and project conventions.

Documentation is built with Hugo and the pinned Docsy theme. Use ./scripts/serve-docs.sh for a local preview and ./scripts/build-docs.sh to produce the deployable site.

9.1 - Development Setup

Local development workflow for infrastructure changes and the documentation site.

Getting started

Follow the Dev Container guide to set up your development environment. The aibox Dev Container includes OpenTofu, Ansible, cloudflared, Hugo, Node.js, Kubernetes clients, and AI assistants.

Documentation development

Preview the documentation site locally:

./scripts/serve-docs.sh
# Open http://localhost:1313

When using a remote Dev Container environment, forward port 1313 from your editor to view the preview.

Build the site:

./scripts/build-docs.sh

The build uses Hugo’s strict template and content validation. The first run installs the pinned Docsy asset dependencies locally and initializes the pinned Docsy theme submodule when needed.

Infrastructure development

If you have a Hetzner Cloud account and want to test infrastructure changes:

cp terraform.tfvars.example terraform.tfvars
# Edit terraform.tfvars with your API token

tofu init
tofu plan    # Preview changes
tofu apply   # Apply changes

Project conventions

See How to Contribute for code conventions and the contribution workflow.

Deploy documentation

To deploy the documentation to GitHub Pages:

./scripts/deploy-docs.sh

This is the standard documentation deployment. It builds the Hugo site locally and pushes the generated public/ directory to the root of the gh-pages branch. GitHub Pages must be configured to serve gh-pages from /; no GitHub Actions workflow is required or used.

The build and deployment scripts create an empty .nojekyll marker in the generated site and at the branch root, so GitHub Pages always serves the prebuilt output directly instead of processing it with Jekyll.

The same script can publish an archived documentation snapshot under a version path. For example:

DOCS_VERSION=v0.1 ./scripts/deploy-docs.sh

Add the corresponding entry to params.versions in hugo.yaml when a release is ready. The versioned build is published below the matching version path and uses that path as its canonical base URL.

9.2 - Code of Conduct

The behavioural standard this project holds contributors to.

This project follows the Contributor Covenant v2.1.

Please read the full text at the link above. In summary: be respectful, be constructive, and create a welcoming environment for everyone.

Reporting

If you experience or witness unacceptable behavior, please contact the project maintainer at bnaard@gmx.net.

Documentation

How it works

Features

Where to start

1 - Project Status

KubeClaw is a prototype

What that means concretely

Why it exists

What it is not

Current state

Versioning

2 - Quick Start

Prerequisites

Accounts

Local Machine

Optional

1. Clone and prepare

2. Open in Dev Container

3. Configure

4. Create infrastructure

5. Set up SSH

6. Connect to the control node

7. Install Cloudflare Tunnel

8. Disable admin node

Next steps

3 - Introduction

3.1 - Architecture

Overview

Node Roles

Master Control Node (control-01, 10.0.0.2)

Replica Control Nodes (control-02+, 10.0.0.3+)

Worker Nodes (10.0.0.x, offset after replicas)

Admin Node (10.0.0.254, temporary)

Network Design

Private Network (10.0.0.0/24)

IPv6-Only

Traffic Flow

Further Reading

3.2 - Security Model

Firewall Rules (Hetzner)

Host-Level Security

Kubernetes-Level Security

3.3 - DNS and NAT64

The Problem: IPv4 Internet from IPv6-Only Nodes

How DNS64 + NAT64 Works (Node Level)

DNS Inside the Kubernetes Cluster (Dual-Stack)

CoreDNS: The Bridge Between Cluster and External DNS

How Pods Reach External Services (Dual-Stack + NAT64)

Full DNS Resolution Example: Pod Resolves an External Name

Full DNS Resolution Example: Host Resolves a Cluster Service

Summary: Who Resolves What

Configuration

For existing nodes

Disabling NAT64

Verification

Technical Details

What cloud-init configures

What the Ansible playbook configures

Worker node specifics

4 - Guide

4.1 - Dev Container

What the Dev Container Provides

Step 1: Prepare Persistent Directories

Step 2: Open in Dev Container

Step 3: Start SSH Agent

Step 4: Verify Setup

4.2 - Infrastructure (OpenTofu)

Step 1: Configure terraform.tfvars

SSH key options

Step 2: Provision Infrastructure

Step 3: Set Up SSH Access

Next Steps

Related

4.3 - Cloudflare Tunnel Setup

Why Cloudflare Tunnel?

Prerequisites

Step 1: Open Zero Trust Dashboard

Step 2: Create a Tunnel

Step 3: Copy the Tunnel Token

Step 4: Add a Public Hostname for SSH