Cloudflare Cuts 70 GB/s From Data Pipeline With Schema, Compression and Sampling Overhaul

Cloudflare's Data team, which operates the pipeline moving data from the company's global network to central data centers, has detailed a series of optimizations that removed roughly 70 GB/s of compressed traffic from its backbone between spring and fall of this year. The team delivers data at about two billion events per second and faces exponential traffic growth against linear hardware growth.

Schema split and compression

In 2022, Cloudflare dismantled its single monolithic message schema, which had packed every product's fields into one stream and forced internal consumers to ingest everything before filtering. Replacing it with product-specific schemas cut backbone bandwidth usage by 35-40%, equivalent to roughly a year of compressed traffic growth. In 2024, the team switched compression from LZ4 to zstd after confirming CPU impact was negligible during a gradual rollout. The change reduced compressed traffic throughput by nearly 40%, letting the same infrastructure carry more than 1.5 times the traffic.

Load balancing and early sampling

In spring 2026, Cloudflare rearchitected replication for disaster resilience. Instead of Logfwdr double-writing every non-regional event to two centralized regions, each event now crosses wide-area transit once, with a shared ingest pool and automatic failover modeled on the Logpush service. Cross-data-center replication happens late in the pipeline, after sampling, through the Buftee buffering layer. For migrated datasets this eliminated about 43 GB/s of compressed transit while adding only about 7 GB/s of replication, a net reduction of 36 GB/s. HTTP analytics traffic fell by more than 50%, and Ready-Analytics dropped from about 28 GB/s to 16 GB/s.

In summer 2026, sampling moved into Logfwdr itself. The algorithm shuffles records per key, then keeps a sample equal to the square root of the record count, scaling estimates against the sample rate. Applied to the top three datasets, it at least halved compressed data on the wire, saving 25 GB/s or more than 2 PB per day.

Aggregating earlier

In fall 2026, Cloudflare pushed rollup aggregation into every pipeline stage: Logfwdr instances, centralized receivers, ClickHouse inserters and ClickHouse aggregating table engines. On the HTTP dataset, compressed traffic fell from over 9 GB/s to under 2 GB/s, an approximate 80% reduction. The team also launched Ready-Aggregates, a one-click aggregating analytics product.

Impact

Across the pipeline, load balancing saved about 38 GB/s, sampling about 25 GB/s and aggregation more than 7 GB/s. Ready-Analytics receivers regained roughly 750 CPUs and 3.5 TiB of memory. Without the interventions, that dataset alone would have breached 60 GB/s by September. Overall, throughput now sits about eight months behind its previously projected growth trajectory, and the changes made this year have halved Cloudflare's projected 2031 data volume. Combined with earlier work, the improvements of the past five years are equivalent to deferring two and a half years of hardware expansion.

Sources: Cloudflare