Modern applications generate data continuously.

E-commerce platforms generate orders and customer activity. IoT devices send sensor readings. Financial applications produce transaction events, and cloud applications generate logs and operational metrics.

The challenge is not simply collecting this data. It is making it available for analytics quickly and reliably.

Traditional batch pipelines often introduce delays because data must be collected before processing begins. Streaming architectures help reduce these delays by continuously ingesting events and delivering them to downstream systems.

In this article, we'll explore how to build a real-time data pipeline using:

• Amazon Kinesis Data Streams
• Amazon Data Firehose
• Amazon S3
• Snowflake
• Amazon CloudWatch

We'll follow a practical architecture in which Kinesis handles ingestion, Firehose manages buffering and delivery, Snowflake receives successful records, and S3 stores failed-delivery records for recovery.

1. Understanding the Architecture

The proposed pipeline follows this flow:

 DATA PRODUCERS Applications / IoT / Logs | v Amazon Kinesis Data Streams | v Amazon Data Firehose | +-------+--------+ | | v v Snowflake Amazon S3 Data Warehouse Error / Backup 

Responsibilities of each service

This separation allows application producers to publish events without managing Snowflake connectivity, custom buffering, or destination retry logic.

Architecture note: Kinesis Data Streams and Data Firehose are managed AWS services. They are not customer-managed servers deployed inside a VPC. Configure network connectivity and access controls according to the selected integration.

2. Ingesting Events with Amazon Kinesis Data Streams

Amazon Kinesis Data Streams is a managed service for collecting and storing real-time streaming records.

Imagine an e-commerce application generating these events:

• Customer registration
• Product views
• Cart updates
• Order placement
• Payment transactions
• Shipment status changes

Instead of exporting these events in a daily batch, the application can publish them to Kinesis as they occur.

Example event

{ "event_id": "evt-10001", "event_type": "order_placed", "customer_id": "customer-501", "order_id": "order-9001", "amount": 249.99, "currency": "USD", "event_timestamp": "2026-11-09T08:30:00Z" } 

Publish an event using Python

We can use Boto3 to send records to a Kinesis data stream.

import boto3 import json from datetime import datetime, timezone kinesis = boto3.client( "kinesis", region_name="us-east-1" ) event = { "event_id": "evt-10001", "event_type": "order_placed", "customer_id": "customer-501", "order_id": "order-9001", "amount": 249.99, "currency": "USD", "event_timestamp": datetime.now( timezone.utc ).isoformat() } response = kinesis.put_record( StreamName="orders-stream", Data=json.dumps(event).encode("utf-8"), PartitionKey=event["customer_id"] ) print("Sequence number:", response["SequenceNumber"]) print("Shard ID:", response["ShardId"]) 

The producer sends:

1. Stream name
2. Event payload
3. Partition key

The partition key determines which shard receives the record.

Understanding Kinesis shards

Kinesis distributes records across shards. Each shard provides a defined amount of read and write capacity.

For provisioned capacity, the documented base limits are:

Actual throughput depends on record sizes, consumer configuration, aggregation, and capacity mode.

For higher workloads, evaluate on-demand capacity or provisioned shard scaling.

3. Amazon Data Firehose: Managed Delivery

After records enter Kinesis, the next component is Amazon Data Firehose, previously known as Amazon Kinesis Data Firehose.

Firehose is a managed delivery service that reads records from supported sources, buffers them, and delivers them to configured destinations.

In our architecture:

• Source: Kinesis Data Streams
• Destination: Snowflake
• Backup location: Amazon S3

Why use Firehose?

Without a managed delivery service, developers may need to build custom consumers to handle:

• Reading records
• Batching events
• Buffering
• Destination connectivity
• Retry logic
• Monitoring
• Failure handling

Firehose manages much of this delivery workflow.

Configuration workflow

A typical setup involves:

1. Create a Kinesis data stream.
2. Create a Firehose delivery stream.
3. Select Kinesis Data Streams as the source.
4. Configure the source stream.
5. Select Snowflake as the destination.
6. Configure Snowflake authentication and target table details.
7. Configure an S3 backup location.
8. Set up monitoring and logging.

Important: When Kinesis Data Streams is configured as the Firehose source, producers write to Kinesis using its APIs, such as PutRecord or PutRecords. The Firehose PutRecord and PutRecordBatch APIs are not used for that source configuration.

Firehose starts reading from the LATEST position when configured with a Kinesis data stream source. Plan your initial setup carefully if historical records must also be processed.

4. Understanding Buffering

Buffering is a key concept in streaming data delivery.

Instead of sending every small record independently, Firehose collects incoming records before attempting delivery.

Two main buffering hints are available:

• Buffer size: The amount of data collected.
• Buffer interval: The maximum time Firehose waits before attempting delivery.

For example, you might configure a 5 MiB buffer size and a 60-second interval.

These are configuration hints, not guaranteed delivery times. Firehose may use different values depending on the destination.

Why does buffering matter?

Imagine an application generating 100 small events every second.

Sending each event separately can create unnecessary request overhead.

Buffering groups records and helps manage delivery more efficiently.

Snowflake-specific behavior

The Snowflake destination has its own buffering behavior.

AWS documents that Firehose internally buffers Snowflake-bound records for approximately one second by default and flushes records to the Snowflake table at that frequency.

The integration also provides a CloudWatch metric for measuring the time records take to be committed to Snowflake.

The input must be a valid JSON object. JSON arrays are not supported as record payloads for this destination.

5. Partitioning: Kinesis vs. Firehose

Partitioning is important, but there are two different concepts in this architecture.

A. Kinesis partition keys

Kinesis uses partition keys to determine which shard receives a record.

For example:

PartitionKey="customer-501" 

Records with the same partition key are routed to the same shard, preserving their order within that shard.

Choosing a good key matters.

If most events use one partition key, traffic may become concentrated on a single shard, creating a hot-shard problem.

Choose keys that balance traffic distribution with ordering requirements.

B. Firehose dynamic partitioning for S3

Firehose supports dynamic partitioning for records delivered to S3.

It can extract values from JSON records and use them to organize objects into S3 prefixes.

Example event:

{ "event_type": "order_placed", "region": "us-east-1", "event_date": "2026-11-09", "customer_id": "customer-501", "amount": 249.99 } 

Possible S3 structure:

s3://company-streaming-data/ region=us-east-1/ event_date=2026-11-09/ records.json 

This organization can make downstream queries easier with Amazon Athena.

Dynamic partitioning is an S3 delivery feature. It is not the same as Kinesis shard partitioning or Snowflake table partitioning.

Avoid over-partitioning

Too many small partitions can create many small files and increase processing overhead.

Useful S3 partition keys often include:

• Event date
• Region
• Event category
• Business unit

Avoid highly unique values such as individual transaction IDs unless there is a specific requirement.

Also remember that Firehose maintains buffers for active dynamic partitions. AWS documents a default active-partition limit of 500 per stream, with a quota-increase process available.

6. Integrating Snowflake

Snowflake is a cloud data platform used for data warehousing, analytics, and data engineering.

In our architecture, Snowflake is the primary destination for successfully delivered streaming records.

Firehose integrates with Snowflake through its streaming ingestion capabilities.

Configuration requirements

You typically need:

• Snowflake account URL
• Snowflake user
• Private-key authentication details
• Database
• Schema
• Target table
• Appropriate permissions
• S3 backup configuration

For security, use AWS Secrets Manager to store Snowflake credentials instead of hardcoding them in application configuration.

Check the current regional availability and integration requirements before choosing your AWS Region.

Create a Snowflake table

One option is to store incoming JSON events in a VARIANT column.

CREATE DATABASE STREAMING_DB; CREATE SCHEMA STREAMING_DB.RAW; CREATE TABLE STREAMING_DB.RAW.ORDER_EVENTS ( EVENT_DATA VARIANT ); 

The VARIANT data type supports semi-structured data and is useful when event schemas evolve.

Query streaming records

After records are delivered, you can extract fields using SQL:

SELECT EVENT_DATA:order_id::STRING AS ORDER_ID, EVENT_DATA:customer_id::STRING AS CUSTOMER_ID, EVENT_DATA:amount::NUMBER(10,2) AS AMOUNT, EVENT_DATA:event_timestamp::TIMESTAMP_TZ AS EVENT_TIME FROM STREAMING_DB.RAW.ORDER_EVENTS; 

Depending on the integration configuration, JSON keys can also be mapped to individual Snowflake columns.

What can you do with the data?

Once data reaches Snowflake, it can support:

• Real-time operational analytics
• Sales and transaction reporting
• Customer behavior analysis
• BI dashboards
• Data science and machine learning
• Historical trend analysis

The streaming pipeline provides a managed path from application events to analytical workloads.

7. Amazon S3: Failed Delivery and Recovery

A production pipeline needs a recovery strategy.

What happens if Snowflake becomes temporarily unavailable?

What if a record cannot be delivered?

What if the payload contains invalid JSON?

This is where Amazon S3 becomes important.

For a Firehose stream using Snowflake as its destination, configure an S3 backup location for error data.

Example structure:

s3://company-streaming-backup/ snowflake-errors/ year=2026/ month=11/ day=09/ failed-records/ 

The exact prefix depends on your Firehose configuration.

How does failure handling work?

Firehose retries destination delivery according to the configured retry duration.

If the retry period expires before successful delivery, the service backs up the relevant data to the configured S3 location.

This gives engineers a place to investigate and recover failed records.

Why are backup records useful?

They help teams:

• Investigate malformed events
• Identify connectivity problems
• Reprocess records after fixing an issue
• Maintain an audit trail
• Monitor data quality
• Recover from temporary downstream failures

Remember: writing failed records to S3 does not automatically replay them into Snowflake.

A separate recovery workflow is required to validate and redeliver those records safely.

Recommended S3 controls

For production environments, consider:

• Server-side encryption with AWS KMS
• Restricted IAM access
• Lifecycle policies
• Versioning where appropriate
• CloudTrail data events when required
• Monitoring backup activity and storage growth

8. Monitoring and Troubleshooting

A streaming pipeline needs continuous monitoring.

Amazon CloudWatch provides metrics and logs that help identify ingestion and delivery problems.

Common troubleshooting scenarios

Records are not reaching Firehose

Check the source stream ARN, IAM permissions, Kinesis incoming-record metrics, and Firehose source-read metrics.

Records are not reaching Snowflake

Check the account URL, authentication, user permissions, destination configuration, and CloudWatch logs.

Invalid JSON records

Validate producer payloads. Send individual JSON objects rather than JSON arrays for the Snowflake destination.

Unexpected delivery delays

Review buffering, source throughput, retry activity, and Snowflake commit metrics.

S3 contains failed records

Inspect error details and determine whether the issue is caused by payload formatting, authentication, destination availability, or configuration.

9. Production Best Practices

Connecting the services is only the beginning. A production-ready pipeline needs careful engineering.

Use a consistent event schema

Include stable identifiers, timestamps, event types, and schema versions.

{ "event_id": "evt-10001", "event_type": "order_placed", "schema_version": "1.0", "event_timestamp": "2026-11-09T08:30:00Z", "source": "checkout-service", "payload": { "order_id": "order-9001", "amount": 249.99 } } 

Choose partition keys carefully

For Kinesis, balance traffic distribution with ordering requirements.

For S3 dynamic partitioning, choose fields that match common analytical filters.

Secure credentials

Use IAM roles for AWS service access and AWS Secrets Manager for Snowflake credentials.

Never hardcode sensitive credentials in source code.

Design for retries and duplicates

Retries can create duplicate-delivery scenarios. Use stable event IDs and downstream deduplication where required.

Monitor data freshness

Measure the time between event creation and successful delivery to Snowflake.

Control storage costs

Use S3 lifecycle policies to manage older backup objects and choose storage classes based on recovery requirements.

Test failure scenarios

Test invalid JSON, expired credentials, destination outages, throttling, and recovery workflows before production deployment.

10. When Should You Use This Architecture?

This architecture is useful when applications need managed streaming ingestion and a delivery path into Snowflake.

It is particularly useful when teams want to avoid operating custom consumers and delivery infrastructure.

For complex transformations, multiple independent consumers, or advanced event processing, consider adding dedicated Kinesis consumers or other processing services.

Conclusion

Building a real-time data pipeline does not necessarily require a large collection of custom applications.

Amazon Kinesis Data Streams provides the ingestion layer. Amazon Data Firehose manages buffering and delivery. Amazon S3 provides a recovery location for failed records. Snowflake makes successfully delivered events available for analytics.

Together, these services create a managed path from application events to analytical insights.

The most important engineering considerations are choosing suitable partition keys, understanding buffering behavior, validating event formats, securing credentials, monitoring delivery, and planning for recovery.

Real-time analytics starts with reliable ingestion — but a production-ready pipeline depends on reliable delivery, observability, and recovery as well.

References

• Amazon Kinesis Data Streams Documentation
• Configure Kinesis Data Streams as a Firehose Source
• Amazon Data Firehose Delivery
• Configure Snowflake as a Firehose Destination
• Firehose Buffering
• Firehose Dynamic Partitioning
• Firehose Troubleshooting
• Amazon S3
• Snowflake Documentation

Have you built a streaming pipeline using Kinesis, Firehose, or Snowflake? Share your architecture and lessons learned in the comments.

AWS #AmazonKinesis #AmazonS3 #Snowflake #DataEngineering #RealTimeAnalytics #StreamingData #CloudComputing