Cloudflare adds SQL Transformers to Logpush export pipeline

What Transformers do

Cloudflare has made Logpush available on all self-serve plans and has generally released Transformers for the same export workflow. Observability teams can now filter records, redact sensitive values, and reshape fields before logs leave Logpush for SIEMs, analytics platforms, or object storage. Filtering before delivery can also reduce downstream data volume and cost because teams no longer need to export, ingest, and store records that are discarded after arrival.

A Transformer is a SQL query written for one Logpush dataset. Logpush applies the query to each batch before export, removes records that do not match the WHERE clause, and uses the SELECT output as the exported NDJSON record. Teams can add, rename, remove, or nest fields, including adding a static provider field to normalize records from multiple CDNs.

How execution works

Customers create, preview, and manage Transformers in Transformer Studio or through the API, then attach a saved version to a compatible NDJSON Logpush job using the same dataset. Each update creates a new version. Cloudflare stores the SQL query in R2 and Bufstream retrieves it through a multi-layer cache, prepares a reusable DataFusion logical plan once per job time slice, and reuses it for every Arrow batch in that slice.

Bufstream receives compressed Cap'n Proto batches from Buftee, decompresses them, parses eligible logs, and builds a typed Apache Arrow RecordBatch before execution. Apache DataFusion and Arrow were selected because DataFusion already powers Cloudflare Basin. The Go service exchanges Arrow IPC buffers with the Rust engine across an FFI boundary exposed through a C ABI and called with cgo. Go passes the input buffer pointer only for the cgo call, Rust owns the output buffer, and Go copies that output into Go-managed memory before releasing the Rust allocation.

Delivery, health, and setup

Transformers are designed to fail closed. If a transform cannot execute, Bufstream neither advances the job delivery offset nor sends the untransformed batch, and it re-attempts failed transforms instead. It progressively delays retries, reports errors on the job, and disables jobs that stay in a failing state for an extended period. The Transformer Health view shows successful and failed transformations, data volume before and after transformation, and transformation duration.

After execution, Bufstream converts Arrow output to NDJSON, batches and gzip-compresses the logs, and uploads them to the configured destination while adjusting upload goroutines within destination concurrency limits. The company said the broader data pipeline ingests more than one billion events per second and Bufstream delivers more than 110 million Logpush records per second. Transformers are available on Free, Pro, Business, and Enterprise plans.

Sources: The Cloudflare Blog