Pipe Dreams

Any engineer can draw a data pipeline. Making one survive contact with production is a different craft entirely.

On a whiteboard, a telemetry pipeline is the work of a moment. Collect events, ship them somewhere, query them later: three boxes and two arrows. The marker is barely dry before everyone nods and moves on.

Then it meets reality. Events arrive out of order. Schemas drift between client versions like accents between generations. A single noisy service floods the queue and starves everything downstream. The tidy diagram, it turns out, was the easy part.

What follows is the architecture behind a telemetry pipeline that ran on AWS, Amazon's cloud platform, and carried millions of events a day without sacrificing either freshness or accuracy. It is not glamorous. It is, however, the sort of thing that keeps working at three in the morning.

The shape of the thing

Producers → API Gateway → Kinesis Data Streams → Lambda (Enrichment) → S3 + DynamoDB
                                                        ↓
                                               Kinesis Firehose → S3 (raw archive)

1. Ingestion: API Gateway + Kinesis

The front door is a plain REST endpoint sitting behind API Gateway, Amazon's managed request-routing layer. It accepts batched event payloads and writes them straight into a Kinesis Data Stream, Amazon's real-time streaming service. The trick is PutRecords, which batches up to 500 events per request and, in doing so, slashes the per-call overhead.

import boto3, json

kinesis = boto3.client('kinesis', region_name='us-east-1')

def put_events(events: list[dict], stream_name: str):
    records = [
        {
            'Data': json.dumps(event).encode(),
            'PartitionKey': event.get('device_id', 'default'),
        }
        for event in events
    ]
    return kinesis.put_records(Records=records, StreamName=stream_name)

Partition keys are not a detail to wave away. Using device_id spreads load evenly across shards and keeps each device's events in order within its shard. Choose carelessly and you get hot shards, cold shards and a great deal of avoidable grief.

2. Enrichment: the Lambda in the middle

A Lambda function—Amazon's serverless compute, billed by the millisecond—reads from the stream, dresses each event with metadata pulled from DynamoDB, Amazon's managed NoSQL store (device registry, tenant information), and fans the results out to their downstream homes.

import boto3, json

dynamodb = boto3.resource('dynamodb')
registry = dynamodb.Table('DeviceRegistry')

def handler(event, context):
    enriched = []
    for record in event['Records']:
        payload = json.loads(record['kinesis']['data'])
        device = registry.get_item(Key={'device_id': payload['device_id']}).get('Item', {})
        enriched.append({**payload, 'tenant_id': device.get('tenant_id'), 'region': device.get('region')})
    # fan out to S3, DynamoDB, alerting...
    return {'statusCode': 200, 'processed': len(enriched)}

Three tuning choices earn their keep:

• Batch size: 100 records. Bigger batches lift throughput but raise the cold-start cost paid on every retry.
• Bisect on error: enabled. It stops one poisoned record from wedging an entire shard.
• Parallelisation factor: 2 per shard. Doubles throughput without over-provisioning.

3. Storage: S3 and DynamoDB, each to its purpose

Hot queries—the last seven days, by device—land in DynamoDB with a 30-day TTL. Cold and analytical queries go to S3, Amazon's object store, delivered by Kinesis Firehose, its managed load-and-deliver service, and partitioned by year/month/day/hour so that Athena, Amazon's serverless query engine, can read them cheaply. Storage is not one problem but two, and pretending otherwise is expensive.

Where it gets interesting

Schema evolution. Every event type carries its version in the payload ("schema_version": "2"), and a schema registry lives in S3. The enrichment Lambda validates and normalises before anything is written downstream. Unknown versions are routed to a dead-letter Firehose bucket for manual review, rather than silently corrupting the record of what happened.

Backpressure. When DynamoDB write capacity spikes, the Lambda throttles. The remedy is a buffer: SQS, Amazon's managed queue, sits between the Lambda and the DynamoDB writes, absorbing bursts and smoothing the write rate on its own.

Monitoring. CloudWatch Metric Filters, part of Amazon's observability suite, scrape the Lambda logs to emit custom metrics for schema_validation_failures and enrichment_errors. A CloudWatch alarm watches them and pages the on-call engineer the moment the error rate clears 1%.

The bill

At 50M events a day across 10 shards, the monthly reckoning looks like this:

The dominant lever is the shard count. Size it to throughput—1 MB/s or 1,000 records/s per shard—and keep an eye on GetRecords.IteratorAgeMilliseconds. Should it creep above a few seconds, add shards. The metric is, in effect, the pipeline quietly telling you it is falling behind.

What to remember

• Batch with PutRecords at ingestion. It is a tenfold cost win over individual puts, and free of charge to adopt.
• Version your event schemas from the first day. Retrofitting them later is a penance nobody enjoys.
• Treat bisect-on-error in Lambda as non-negotiable for a production stream.
• Keep SQS as a write buffer. It is cheap insurance against downstream throttling.

The full Terraform module for this stack is something I am planning to open-source. More on that soon. Three boxes and two arrows, after all, were never going to be the whole story.