PDF, ZIP and binary files streaming over Kafka with names intact.
Day 07 of the WKafka Open-Source Engineering Series.
Don't provision a cloud bucket when all you need is sending small PDFs and ZIPs down your pipeline. WKafka supports format='file' out of the box.
The Pain Points We Faced
- Building S3/MinIO upload/download pipelines just to move 500KB report files
- Lost original filenames and mime-types when converting files to raw bytes
- High latency polling for cloud object storage propagation
The Implementation
# Producer kafka.produce_file(topic="reports", file_path="audit_2026.pdf") # Consumer @kafka.consumer(topic="reports", format="file") def on_report(msg): msg.save_file(target_dir="received_pipeline_files/") Why This Architecture Wins
- format='file': Native file serializer bundles filename and binary data.
- msg.save_file(): One-liner to write the incoming file to disk preserving name.
- Sub-Second Delivery: Direct broker streaming without external storage bottlenecks.
Verification & Status
Tested and verified with Apache Kafka against real broker clusters (see EXAMPLES_STATUS.md in repository). Compatible with Python 3.9 through 3.14 with strict typing.
