Key points
- Logpush Transformers use Apache DataFusion and Arrow to process SQL queries on log batches before export.
- Teams can filter out health checks, redact sensitive fields, and normalize schemas for analytics platforms.
- The feature is generally available on all self-serve plans, reducing the need for separate processing services.
Cloudflare has made its Logpush feature generally available on all self-serve plans, introducing SQL-based Transformers that allow organizations to modify log data before it leaves their environment. This update enables security and observability teams to filter, redact, and reshape log streams using standard SQL queries. By processing data within the export pipeline, companies can reduce downstream storage costs and ensure that only necessary information reaches external destinations like SIEMs or object storage.
In plain English
Previously, exporting logs often required separate processing services that teams had to deploy, scale, and maintain. With Transformers, this processing happens inside Cloudflare’s export workflow. Teams write a SQL query that defines which records to keep, how to rename or nest fields, and what sensitive data to remove. Logpush applies this query to each batch of logs before exporting them in NDJSON format. This means destinations receive data in the exact structure needed for dashboards or investigations, without extra handling.
The background
Cloudflare’s data pipeline ingests more than one billion events per second, with Logpush delivering over 110 million records per second to customer destinations. The system runs across hundreds of Kubernetes pods, distributing work horizontally. Most jobs operate at lower volumes, while a small number process millions of logs each second. Transformers were designed to meet these high-throughput requirements while maintaining reliable delivery and predictable resource use. They are powered by Apache DataFusion and Apache Arrow, ensuring efficient processing without introducing additional failure scenarios.
What changes now
Teams can now create, preview, and manage Transformers through Transformer Studio in the Cloudflare dashboard or programmatically via the API. Each saved Transformer creates a new version, allowing users to track changes over time. Customers can attach a Transformer to a compatible NDJSON Logpush job that uses the same dataset. This capability allows for schema normalization across multiple CDN providers, redaction of client IP addresses, and filtering of internal traffic, all before the data is exported.
What to do and how to stay safe: Cloudflare
- Review current log export jobs to identify sensitive fields that should be redacted before leaving your environment.
- Define SQL queries to filter out non-essential traffic, such as health checks, to reduce storage and ingestion costs.
- Test Transformers in a staging environment to ensure data reshaping meets your analytics or SIEM schema requirements.
- Monitor resource usage after enabling Transformers to ensure predictable performance within your export workflow.
Step-by-step guide: Smartphone Security Updates for Small Business Operations
General security guidance from the Patch Review newsroom. It is not confirmed advice from the organisations named in this story.
Frequently asked questions
What is the technical basis for Logpush Transformers?
Transformers are powered by Apache DataFusion and Apache Arrow, embedded in Logpush's high-throughput export pipeline.
Can I manage Transformers programmatically?
Yes, customers can create and manage Transformers through the Cloudflare dashboard or via the API.
Does this feature affect export costs?
Filtering records before delivery can reduce downstream data volume, potentially lowering costs for export, ingestion, and storage.
