Cloudflare Cuts DNS Zone Rebuild Times by Half and RAM Use by 85%

Cloudflare says its engineers have optimized DNS zone rebuild services, halving rebuild times and cutting memory use by 85 percent across its network. The work follows earlier memory savings and targets a DNS fleet handling 800 zone changes per second.

Background and scale

Cloudflare's DNS fleet handles 800 zone changes per second, up from 250 four years ago. A DNS zone holds records, settings, and metadata for a domain, including IP addresses, hostnames, text lines, subdomains, redirections, and spam or certificate information. The company says consistency must be maintained across thousands of DNS servers, and some domains have more than 20 million records. Earlier, engineers found 100 TB of memory savings in the DNS cache and another 100 TB by improving task-server routing. The company had already optimized record updates in 2022 by only updating needed data.

Partial rebuilds for settings

Previously, setting updates required full builds. Engineers added reasons and extra information to zone change events, allowing those events to be classified. Specific setting changes can then use smaller builds. Apex and metadata changes, such as Universal SSL, form one type. Changing a domain's nameserver or wildcards triggers another smaller procedure. Full builds remain for cases without a partial build mechanism. Domains with many records are now updated piecemeal. The company reports an 85 percent drop in RAM use for Zone Builder servers, from a range of 50 to 79 GB to close to 7 GB, mostly from these changes.

Scheduler and concurrency

The second optimization combined two separate planning and implementation schedulers into one. The new scheduler detects multiple changes affecting the same zone and combines them. Cloudflare uses Apache Kafka for event messaging. Previously, when a message triggered a slow build, the scheduler waited for that job to finish and did not process requests arriving in the meantime, because Kafka advances messages in sequence. Now it acknowledges and runs incoming smaller jobs that arrive during that time. Requests are fulfilled soon after being issued in a consistent cadence, rather than in bursts after a long queue. The team also tightened queue processing intervals. Median build time fell from 7.7 seconds to 3.3 seconds.

Sources: Tom's Hardware