Go With the Flow

Streaming data is easy to start and hard to steer. The architecture, the pricing and the small operational print are where the choices bite.

A sensor on a factory floor emits a reading. A shopper taps a button. A trade clears. Each is a single, perishable event, and each is worthless if it arrives too late to act on. The old habit was to gather such events into a pile and process the pile overnight. The modern habit is not to let them settle at all.

That is the promise of Amazon Kinesis Data Streams, the streaming service from Amazon's cloud arm, Amazon Web Services (AWS). When you need to ingest and process streaming data in real time — telemetry from IoT devices, clickstream events, application logs, financial transactions — it is the AWS-native answer. It stores streamed data synchronously across three Availability Zones, supports data retention up to 365 days, and integrates natively with Lambda, Flink, and Firehose.

AWS describes the architecture as a stack of five logical layers, each composed of purpose-built components:

1. Streaming Sources      (IoT devices, applications, services)
2. Stream Ingestion       (Kinesis Data Streams, IoT Core)
3. Stream Processing      (Lambda, Kinesis Data Analytics / Flink)
4. Serving Layer          (DynamoDB, OpenSearch, Timestream)
5. Consumption Layer      (dashboards, alerts, downstream systems)

Each layer can scale independently. That is the core advantage over the batch-oriented pipelines it replaces, and it is not a small one.

Pay by the hour, or by the drink

Kinesis offers two capacity modes. The choice affects cost, scaling behaviour and operational overhead:

The sensible default is to begin with On-Demand and let it absorb the guesswork, sparing you the twin sins of under- and over-provisioning. Move to Provisioned once you have a clear throughput baseline; at steady state it is cheaper. And watch the meter. Per AWS best practices, monitor GetRecords.IteratorAgeMilliseconds — if it climbs consistently, you need more shards or faster consumers.

Getting data in

The first task is to put events onto the stream. The blunt approach is a direct put from the producers themselves:

import boto3, json

kinesis = boto3.client('kinesis', region_name='us-east-1')

def publish_events(events: list[dict], stream: str):
    records = [
        {
            'Data': json.dumps(e).encode(),
            'PartitionKey': e.get('device_id', 'default'),
        }
        for e in events
    ]
    # PutRecords batches up to 500 records per call — much cheaper than individual PutRecord
    return kinesis.put_records(Records=records, StreamName=stream)

The partition key does quiet but consequential work: it determines how records fan out across shards. A high-cardinality key such as device_id or user_id distributes load evenly and keeps per-entity records ordered within a shard. A low-cardinality key — a single constant, say — sends everything to one shard and creates a hot partition. The lesson is old but easily forgotten: pick your key badly and the cleverest pipeline collapses into a queue of one.

Then there is scale. For fleets of sensors, AWS recommends routing IoT device data through AWS IoT Core first. IoT Core handles device authentication, certificate management, and MQTT protocol termination, then fans data into Kinesis. This avoids putting AWS credentials on embedded devices — a precaution whose wisdom becomes obvious the first time a device is stolen.

Lambda, on the other end

At the consuming end, Lambda, Amazon's serverless compute service, integrates natively with Kinesis as an event source. It arrives with a batch of records and expects you to deal with them:

# Lambda handler — receives a batch of Kinesis records
def handler(event, context):
    for record in event['Records']:
        payload = json.loads(record['kinesis']['data'])
        process(payload)
    return {'statusCode': 200}

The defaults will not save you in production. Four settings matter:

• Bisect on error: splits a failing batch in half and retries each half separately, isolating bad records without blocking the whole shard
• Maximum retry attempts: set a finite limit; otherwise a poison-pill record blocks a shard indefinitely
• Destination on failure: route failed records to SQS or S3 for later inspection
• Parallelization factor: 2–10 concurrent Lambda invocations per shard for higher throughput

And then there is idempotency, the least glamorous virtue in distributed systems. Kinesis guarantees at-least-once delivery — producers may retry, and Lambda retries on error. Your consumer must therefore handle duplicate records. Use a deduplication key (e.g., event UUID in DynamoDB with a TTL) if downstream systems can't absorb duplicates. The alternative is a database that quietly double-counts, which is worse than one that fails loudly.

The trouble with clocks

Streaming pipelines tend to assume the world agrees on the time. It does not. Device clocks drift. AWS calls out that IoT devices may emit timestamps more than 15 minutes off — in either direction — due to internal clock errors or daylight savings transitions. If you're windowing on event timestamps in Flink or Kinesis Data Analytics, build in a late-data tolerance and use the Kinesis approximateArrivalTimestamp as a fallback when device timestamps look implausible. A cheap sensor's opinion of the hour is not to be trusted.

Firehose keeps the receipts

Real-time is not the whole story; someone always wants the history. Firehose, a companion delivery service, sits alongside Kinesis Data Streams and handles the durable archive path with zero consumer code. It buffers records and delivers them to S3 (or OpenSearch, Redshift, Splunk) in configurable batches:

Kinesis Data Streams → Firehose → S3 (partitioned by year/month/day/hour)

S3 output partitioned by time is immediately Athena-compatible — you get ad-hoc SQL over months of raw event history with no ETL required. The stream keeps the present moving; Firehose quietly banks the past.

What to watch

A streaming pipeline that no one is watching is a pipeline that has already failed silently. The metrics that matter:

Set CloudWatch alarms on IteratorAgeMilliseconds > 60,000ms (1 minute behind) — that's the signal to add shards or optimise your consumer before lag compounds. Lag, left alone, does not stay small.

The unglamorous baseline

Security is the part everyone means to get to later. Kinesis Data Streams supports server-side encryption with AWS KMS out of the box — enable it for any stream that carries sensitive data. For network isolation, use VPC interface endpoints so traffic between your VPC and Kinesis never traverses the public internet. Scope IAM policies to specific stream ARNs — producers should only have kinesis:PutRecords, consumers only kinesis:GetRecords and kinesis:GetShardIterator.

None of it is difficult. Which is precisely why it is so often skipped. Data in motion is a fine thing, right up until it moves somewhere it should not.

Sources:

• Architectural Patterns for Real-Time Analytics with Kinesis Data Streams, Part 1 — AWS Big Data Blog
• Build and Optimize a Real-Time Stream Processing Pipeline with Kinesis and Flink — AWS Big Data Blog
• Real-Time Data Processing Pipeline with Kinesis and Lambda — Simple AWS Newsletter
• Near Real-Time Processing with Kinesis, Timestream, and Grafana — AWS Database Blog