Feeding the OLAP beast: ClickHouse massive bulk ingestion steps.

ClickHouse is a columnar beast, but treating it like MySQL by inserting row-by-row will bring your cluster to its knees. ClickHouseBulkStep packages optimal bulk buffering into a reusable step.

Here is how you use ClickHouseBulkStep & OLAP Pipelines in a production pipeline with wpipe-steps:

from wpipe import Pipeline from wpipe_steps.database import ClickHouseBulkStep pipeline = Pipeline(pipeline_name="telemetry_ingest") pipeline.set_steps([ ClickHouseBulkStep.as_step( name="insert_telemetry_batch", host="clickhouse.production.internal", database="analytics", table="device_telemetry", data_key="telemetry_batch", response_key="clickhouse_status" ) ]) sample_batch = [ {"sensor_id": f"S-{i}", "temp": 24.5 + i, "timestamp": "2026-09-19 12:00:00"} for i in range(10000) ] result = pipeline.run({"telemetry_batch": sample_batch}) print(f"Inserted {result['clickhouse_status']['count']} records successfully!")

Why developers love wpipe-steps:

  • 196 cataloged steps covering Redis, ClickHouse, MySQL, WAF, S3, Docker, and local HuggingFace AI.
  • Lazy-loading imports for instant sub-100ms startup times.
  • Clean .as_step() factory interface.

Check out the full repository on GitHub!