Start here to understand the design decisions behind KubeClaw before provisioning infrastructure.
This is the multi-page printable view of this section. Click here to print.
Introduction
Understand KubeClaw’s architecture, network model, and security boundaries.
1 - Architecture
Node roles, private network layout, and how traffic reaches an IPv6-only cluster.
Overview
KubeClaw creates a secure, IPv6-only Kubernetes cluster on Hetzner Cloud. The design prioritizes security through network isolation: no public IPv4 addresses, SSH access exclusively via Cloudflare Tunnel, and per-namespace egress control with Cilium network policies.
┌─────────────────────────────────────────────────────────────────┐
│ Internet │
└─────────────────────────────────────────────────────────────────┘
│ │
▼ ▼
┌──────────────┐ ┌───────────────┐
│ Cloudflare │ │ Admin Node │
│ Tunnel │ │ 10.0.0.254 │
│ (permanent) │ │ (temporary, │
└──────┬───────┘ │ public IPv6) │
│ └───────┬───────┘
▼ ▼
┌─────────────────────────────────────────────────────────────────┐
│ Hetzner Cloud │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ Private Network (10.0.0.0/24) │ │
│ │ │ │
│ │ ┌─────────────────┐ ┌─────────────────┐ │ │
│ │ │ control-01 │ │ control-02+ │ │ │
│ │ │ 10.0.0.2 │ │ 10.0.0.3+ │ │ │
│ │ │ (master, │◄►│ (replicas, │ │ │
│ │ │ cloudflared) │ │ 0-n instances) │ │ │
│ │ └────────┬────────┘ └─────────────────┘ │ │
│ │ │ │ │
│ │ ▼ │ │
│ │ ┌─────────────────┐ │ │
│ │ │ worker-nodes │ │ │
│ │ │ (0-n instances) │ │ │
│ │ └─────────────────┘ │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
Node Roles
Master Control Node (control-01, 10.0.0.2)
The master control node always exists and serves as:
- Kubernetes control plane – runs etcd, kube-apiserver, kube-scheduler, kube-controller-manager
- Cloudflare Tunnel endpoint – runs
cloudflaredfor SSH access from the internet - SSH gateway – all other nodes are reached through this node
The master node always has public IPv6 (required for cloudflared outbound connections) and accepts SSH from the private network and from localhost (for the Cloudflare Tunnel). The tunnel can be auto-configured via cloudflare_tunnel_token or installed manually.
Replica Control Nodes (control-02+, 10.0.0.3+)
Optional nodes for high-availability control plane. Same configuration as the master, minus cloudflared. IPs start at 10.0.0.3 and increment.
Worker Nodes (10.0.0.x, offset after replicas)
Optional compute nodes for running workloads. Workers have restricted connectivity:
- Inbound: SSH from private network only
- Outbound: DNS (port 53), HTTP (port 80), HTTPS (port 443), internal network only
- No TCP forwarding – prevents workers from being used as jump hosts
Worker IPs start after the last replica control node.
Admin Node (10.0.0.254, temporary)
A temporary jump host with public IPv6, used only during initial setup before the Cloudflare Tunnel is configured. Created by default (enable_admin_node = true) and should be disabled after tunnel setup.
Network Design
Private Network (10.0.0.0/24)
All nodes communicate via a Hetzner private network. IP assignments:
| Address | Node |
|---|---|
10.0.0.2 | Master control node |
10.0.0.3+ | Replica control nodes |
10.0.0.x | Worker nodes (offset after replicas) |
10.0.0.254 | Admin node (temporary) |
IPv6-Only
Nodes have no public IPv4 addresses. The master control node always has public IPv6 (required for cloudflared). Replica control nodes and workers can optionally have public IPv6 disabled via enable_public_ipv6 = false to air-gap them from the internet. NAT64/DNS64 provides transparent IPv4 reachability for accessing IPv4-only services (GitHub, container registries, package repos).
Traffic Flow
- SSH access: Internet → Cloudflare Tunnel → Master control node (localhost:22) → Private network → Other nodes
- Outbound (control nodes): Full outbound connectivity via IPv6 + NAT64
- Outbound (workers): Restricted to DNS, HTTP/S only
Further Reading
- DNS and NAT64 – how IPv6-only nodes reach IPv4 services, CoreDNS configuration, and Kubernetes DNS architecture
- Security Model – firewall rules, SSH hardening, and Kubernetes-level security
2 - Security Model
The three security layers: Hetzner firewalls, host hardening, and Kubernetes network policies.
KubeClaw’s security is layered across three levels: Hetzner Cloud firewalls, host-level hardening, and Kubernetes network policies. See Architecture for the overall network design.
Firewall Rules (Hetzner)
Hetzner Cloud firewalls are the first line of defense. Each node role has its own firewall:
- Control nodes: SSH from private network + localhost (for tunnel), ICMP from private network
- Worker nodes: SSH from private network, ICMP from private network
- Admin node: SSH from anywhere (temporary), ICMP from private network
Kubernetes ports (6443, 10250, 2379-2380, 30000-32767) are intentionally excluded from the Hetzner firewall and added only when deploying Kubernetes.
Host-Level Security
- SSH hardening: key-only auth, no root login, limited retries, no TCP forwarding on workers
- fail2ban: SSH brute-force protection on all nodes
- UFW: host-level firewall enforcing the same rules as the Hetzner firewall
For operational security details (SSH configuration directives, monitoring, security auditing), see Operations: Security.
Kubernetes-Level Security
- Cilium CNI: eBPF-based network policies with FQDN egress filtering
- Namespace isolation:
system-unrestricted(full egress) andapps-restricted(whitelist-only egress) - Container hardening: non-root users, dropped capabilities, resource limits
3 - DNS and NAT64
How DNS64 and NAT64 give IPv6-only nodes and pods transparent access to IPv4-only services.
The cluster is IPv6-only – but many services (GitHub CDN, container registries, package repos) are IPv4-only. NAT64/DNS64 provides transparent IPv4 reachability at the network layer, no application changes needed. This page covers how DNS64/NAT64 works, how it integrates with Kubernetes CoreDNS, and how to configure it.
The Problem: IPv4 Internet from IPv6-Only Nodes
The nodes have no public IPv4 addresses. Most internet services (GitHub, Docker Hub, package repos) have IPv4 addresses. How does an IPv6-only node reach them?
┌──────────────────────────────────────────────────────────────────┐
│ The Problem │
│ │
│ Node (IPv6 only) ──────╳──────► github.com (140.82.121.3) │
│ 2a01:4f8:... │ IPv4 address │
│ No IPv4 route! │
└──────────────────────────────────────────────────────────────────┘
The answer is DNS64 + NAT64, a standard mechanism (RFC 6146/6147) that gives IPv6-only clients transparent access to IPv4 servers.
How DNS64 + NAT64 Works (Node Level)
When a node needs to reach an IPv4-only service, the DNS64 resolver synthesizes a special IPv6 address that embeds the IPv4 address:
┌──────────────────────────────────────────────────────────────────┐
│ Step 1: DNS64 Resolution │
│ │
│ Node DNS64 Resolver Authoritative │
│ ┌──────┐ ┌──────────────┐ DNS Server │
│ │ │──"github.com"─►│ │──"github.com"──►┌─────┐ │
│ │ │ "AAAA?" │ 2001:67c: │ "AAAA?" │ │ │
│ │ │ │ 2b0::4 │◄── no AAAA ─────│ │ │
│ │ │ │ │──"github.com"──►│ │ │
│ │ │ │ │ "A?" │ │ │
│ │ │ │ │◄── 140.82.121.3 ─│ │ │
│ │ │ │ │ └─────┘ │
│ │ │ │ Synthesize: │ │
│ │ │◄───────────────│ 64:ff9b:: │ │
│ │ │ AAAA record: │ 8c52:7903 │ │
│ └──────┘ 64:ff9b:: └──────────────┘ │
│ 8c52:7903 │
│ │
│ The DNS64 resolver embeds the IPv4 address (140.82.121.3 = │
│ 0x8c527903) into the well-known NAT64 prefix 64:ff9b::/96. │
└──────────────────────────────────────────────────────────────────┘
┌──────────────────────────────────────────────────────────────────┐
│ Step 2: NAT64 Translation │
│ │
│ Node NAT64 Gateway github.com │
│ ┌──────┐ (nat64.net) ┌──────────┐ │
│ │ │──IPv6 pkt───►┌──────────────┐──IPv4 pkt─►│ │ │
│ │ │ dst: │ Recognizes │ dst: │140.82. │ │
│ │ │ 64:ff9b:: │ 64:ff9b:: │ 140.82. │121.3 │ │
│ │ │ 8c52:7903 │ prefix, │ 121.3 │ │ │
│ │ │ │ extracts │ │ │ │
│ │ │◄─IPv6 pkt───│ IPv4 addr, │◄─IPv4 pkt──│ │ │
│ └──────┘ │ translates │ └──────────┘ │
│ └──────────────┘ │
│ │
│ The node sends IPv6 traffic to 64:ff9b::8c52:7903. The NAT64 │
│ gateway strips the prefix, extracts 140.82.121.3, and forwards │
│ the packet as regular IPv4. Responses are translated back. │
└──────────────────────────────────────────────────────────────────┘
Key detail: DNS64 only synthesizes AAAA records for domains that have no native AAAA record. If a domain already has an IPv6 address (like google.com), the DNS64 resolver returns the real AAAA record and no synthesis happens.
DNS Inside the Kubernetes Cluster (Dual-Stack)
With dual-stack networking, every pod gets both an IPv4 address (for internal cluster communication) and an IPv6 address (for external access via DNS64/NAT64). This eliminates the need for hostNetwork workarounds:
┌──────────────────────────────────────────────────────────────────┐
│ DNS Consumers and Their Resolvers │
│ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ Host processes (cloudflared, apt, curl) │ │
│ │ /etc/resolv.conf: │ │
│ │ nameserver 10.96.0.10 ◄── CoreDNS ClusterIP │ │
│ │ nameserver 2a01:4ff:ff00::add:2 ◄── Hetzner DNS │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ All pods (CoreDNS, CSI, OpenClaw, nginx, ...) │ │
│ │ /etc/resolv.conf: │ │
│ │ nameserver 10.96.0.10 ◄── CoreDNS ClusterIP │ │
│ │ search apps-restricted.svc.cluster.local │ │
│ │ svc.cluster.local cluster.local │ │
│ │ │ │
│ │ Pod IPs: 10.244.x.x (IPv4) + fd00:10:244::x (IPv6) │ │
│ │ External access: via IPv6 → DNS64/NAT64 → IPv4 internet │ │
│ └─────────────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────────┘
CoreDNS: The Bridge Between Cluster and External DNS
CoreDNS runs as a regular pod with dual-stack addresses. It forwards external queries to DNS64 resolvers, which synthesize AAAA records for IPv4-only domains. This allows all pods to reach external services via NAT64:
┌──────────────────────────────────────────────────────────────────┐
│ CoreDNS Query Flow │
│ │
│ CoreDNS (10.96.0.10) │
│ dual-stack pod │
│ ┌────────────────────┐ │
│ │ │ │
│ Pod queries │ 1. kubernetes │ │
│ "nginx.apps- │ plugin │ │
│ restricted.svc. ───►│ ↓ │ │
│ cluster.local" │ Matches │ ──► 10.111.194.145│
│ │ cluster.local? │ (ClusterIP) │
│ │ YES → respond │ │
│ │ from K8s API │ │
│ │ │ │
│ Pod queries │ 2. forward │ │
│ "github.com" ───►│ plugin │ │
│ │ ↓ │ │
│ │ Matches │ │
│ │ cluster.local? │ │
│ │ NO → forward to │ │
│ │ DNS64 resolvers │──► DNS64 resolver │
│ │ 2001:67c:2b0::4 │ 2001:67c:2b0::4 │
│ └────────────────────┘ ↓ │
│ Synthesizes AAAA │
│ 64:ff9b::8c52:..│
└──────────────────────────────────────────────────────────────────┘
How Pods Reach External Services (Dual-Stack + NAT64)
With dual-stack, pods have IPv6 addresses and can route to the NAT64 prefix. Cilium’s IPv6 masquerading translates pod source addresses to the node’s public IPv6:
┌──────────────────────────────────────────────────────────────────┐
│ Pod Network Connectivity (Dual-Stack) │
│ │
│ Pod (10.244.0.5 + fd00:10:244::5) │
│ ├── ✅ → 10.96.0.10 (CoreDNS ClusterIP) ── DNS works │
│ ├── ✅ → 10.111.194.145 (nginx ClusterIP) ── service routing │
│ ├── ✅ → 10.0.0.2 (node private IP) ── host reachable │
│ ├── ✅ → 64:ff9b::8c52:7903 (github.com via NAT64) ── works! │
│ └── ✅ → 2a00:1450:... (google.com native IPv6) ── works! │
│ │
│ Flow for IPv4-only destinations (e.g., api.anthropic.com): │
│ 1. Pod queries CoreDNS → forwards to DNS64 resolver │
│ 2. DNS64 synthesizes: api.anthropic.com → 64:ff9b::6812:0000 │
│ 3. Cilium DNS proxy records the FQDN→IP mapping │
│ 4. Pod sends IPv6 to 64:ff9b::6812:0000 │
│ 5. Cilium masquerades src to node's public IPv6 │
│ 6. NAT64 gateway translates to IPv4 → reaches api.anthropic.com │
│ │
│ Cilium FQDN policies enforce egress at every step: │
│ toFQDNs: "api.anthropic.com" → allows 64:ff9b::6812:0000 │
│ All other external traffic is DENIED. │
└──────────────────────────────────────────────────────────────────┘
Full DNS Resolution Example: Pod Resolves an External Name
Here’s the complete flow when a pod in apps-restricted queries api.anthropic.com:
┌──────────────────────────────────────────────────────────────────┐
│ Complete DNS Flow: Pod → CoreDNS → DNS64 → NAT64 │
│ │
│ 1. Pod sends DNS query │
│ src: 10.244.0.5 → dst: 10.96.0.10:53 │
│ "What is api.anthropic.com?" │
│ │ │
│ ▼ │
│ 2. Cilium DNS proxy intercepts the query │
│ Records the FQDN for policy matching │
│ Forwards to CoreDNS │
│ │ │
│ ▼ │
│ 3. CoreDNS receives query │
│ kubernetes plugin: "api.anthropic.com" ≠ cluster.local │
│ forward plugin: forward to DNS64 resolver (2001:67c:2b0::4) │
│ │ │
│ ▼ │
│ 4. DNS64 resolver synthesizes AAAA record │
│ api.anthropic.com has only A records (104.18.x.x) │
│ Synthesizes: AAAA 64:ff9b::6812:0000 │
│ │ │
│ ▼ │
│ 5. Cilium DNS proxy records the mapping │
│ api.anthropic.com → 64:ff9b::6812:0000 │
│ toFQDNs rule "api.anthropic.com" now allows this IP │
│ │ │
│ ▼ │
│ 6. Pod connects to 64:ff9b::6812:0000:443 │
│ Cilium checks egress policy → ALLOWED (FQDN match) │
│ IPv6 masquerade: src becomes node's public IPv6 │
│ NAT64 gateway translates to IPv4 104.18.x.x │
│ ✅ SUCCESS -- pod reaches api.anthropic.com │
└──────────────────────────────────────────────────────────────────┘
Full DNS Resolution Example: Host Resolves a Cluster Service
Here’s the flow when cloudflared (system service on the host) needs to reach a Kubernetes service:
┌──────────────────────────────────────────────────────────────────┐
│ Complete DNS Flow: cloudflared → CoreDNS → Kubernetes API │
│ │
│ 1. cloudflared queries the host's DNS │
│ /etc/resolv.conf: nameserver 10.96.0.10 (CoreDNS ClusterIP) │
│ "What is nginx.apps-restricted.svc.cluster.local?" │
│ │ │
│ ▼ │
│ 2. CoreDNS receives query │
│ kubernetes plugin: matches *.svc.cluster.local │
│ Queries Kubernetes API for Service "nginx" in │
│ namespace "apps-restricted" │
│ │ │
│ ▼ │
│ 3. CoreDNS returns ClusterIP │
│ nginx.apps-restricted.svc.cluster.local → 10.111.194.145 │
│ │ │
│ ▼ │
│ 4. cloudflared connects to 10.111.194.145:80 │
│ Cilium routes the ClusterIP to the nginx pod │
│ ✅ SUCCESS -- host can reach ClusterIP via Cilium │
└──────────────────────────────────────────────────────────────────┘
Summary: Who Resolves What
| Consumer | Resolver | Cluster names | External names | External connectivity |
|---|---|---|---|---|
| Host processes (cloudflared, apt) | CoreDNS (ClusterIP) + Hetzner DNS | Yes (via CoreDNS) | Yes (via Hetzner DNS) | Full (IPv6 + NAT64) |
| All pods (CoreDNS, CSI, OpenClaw, etc.) | CoreDNS (10.96.0.10 ClusterIP) | Yes | Yes (AAAA synthesized via DNS64) | Full (IPv6 + NAT64 via masquerade) |
| Node itself (DNS64 configured) | DNS64 resolvers (2001:67c:2b0::4) | No | Yes (AAAA synthesized) | Full (IPv6 + NAT64) |
Configuration
NAT64/DNS64 is enabled by default (enable_nat64 = true). Cloud-init configures it on new nodes automatically. Default resolvers are from nat64.net (Nuremberg, Helsinki, Amsterdam) – close to Hetzner’s fsn1 datacenter.
For existing nodes
Run the Ansible playbook:
cd ansible
ansible-playbook playbooks/configure-nat64.yml
Limit to specific node groups:
ansible-playbook playbooks/configure-nat64.yml --limit control_nodes
Override resolvers:
ansible-playbook playbooks/configure-nat64.yml \
-e '{"dns64_resolvers":["2a01:4f8:c2c:123f::1"]}'
Disabling NAT64
If you set up your own DNS infrastructure:
# terraform.tfvars
enable_nat64 = false
Verification
# DNS64 synthesis (should show AAAA record with 64:ff9b:: prefix)
resolvectl query github.com
# End-to-end connectivity
curl -6 https://github.com
Technical Details
What cloud-init configures
- DNS64 resolvers in
/etc/systemd/resolved.conf.d/dns64.conf - NAT64 route:
64:ff9b::/96via the default IPv6 gateway networkd-dispatcherscript to persist the route across reboots
What the Ansible playbook configures
The same as cloud-init, plus:
- Removes old Hetzner DNS UFW rules on worker nodes
- Adds DNS64 resolver allow rules in UFW (workers only)
- Adds NAT64 prefix UFW rule (workers only)
- Verifies DNS64 resolution and NAT64 connectivity
Worker node specifics
Worker nodes have restricted outbound access. The NAT64 configuration adds:
- UFW rules allowing DNS to DNS64 resolvers (instead of Hetzner DNS)
- UFW rule allowing traffic to the
64:ff9b::/96prefix