Hetzner Details Evolution of Its Cloud Network Stack

From vServers to Cloud

Hetzner's vServer product, launched in 2011, used native Linux bridges and static routes with dual-stack connectivity and up to 1 Gb/s per host, eventually serving around 25,000 instances. A follow-up in September 2015 moved to a hyper-converged Ceph-based setup with BGP dynamic routing, 1:1 NAT for IPv4, and routed IPv6 prefixes. Host links were upgraded to 2x 10 Gb/s in 2016, and the setup reached about 50,000 instances before 2018.

Open vSwitch Takes Over

When Hetzner Cloud launched in 2018, NAT was dropped for direct IPv4 routing. An Open vSwitch-based data plane arrived in July 2019 alongside private cloud networks using VXLAN. The public network migrated to Open vSwitch in November 2020, enabling stateful cloud firewalls based on Open vSwitch flows and netfilter in March 2021.

Custom Orchestration with Flusskrebs

Instead of the standard OVN control plane, Hetzner built its own Python-based orchestrator called Flusskrebs. It uses a REST API to install the Open vSwitch flows each VM or Load Balancer needs, including user-defined firewall rules, and acts as the central DHCP server for private interfaces. Each host also runs a local udhcpd for public interfaces and a metadata server at 169.254.169.254 used by cloud-init and other tools.

Architecture and What's Next

Each host runs a central Open vSwitch bridge connecting all servers, with one public interface and up to three private interfaces per server. Hosts typically use two 10 Gb/s LACP-bonded uplinks, with migration to native routed BGP uplinks underway. Stateful firewalling uses netfilter connection tracking, with the internal ctcount tool capping each server at 80,000 concurrent connections. Hetzner reports the stack now serves over a million cloud servers but has reached scalability limits, prompting work on a fully self-built successor to be detailed in a follow-up article.

Sources: hetzner.com