DestinationsStreamingAmazon Kinesis

Amazon Kinesis

Stream data to Amazon Kinesis Data Streams for real-time processing. Use SignalSmith to send model results, audience membership changes, or enriched records to Kinesis for consumption by Lambda functions, analytics pipelines, or other AWS services.

Prerequisites

  • An Amazon Kinesis Data Stream
  • An IAM user or role with kinesis:PutRecord and kinesis:PutRecords permissions on the target stream
  • AWS access key ID and secret access key for authentication

Authentication

Kinesis uses AWS Access Keys authentication.

  1. In the AWS IAM Console, create an access key for a user with Kinesis write permissions
  2. Enter the Access Key ID and Secret Access Key in SignalSmith

Required IAM Permissions

The IAM user needs the following permissions on the target stream:

{
  "Effect": "Allow",
  "Action": [
    "kinesis:PutRecord",
    "kinesis:PutRecords",
    "kinesis:DescribeStream"
  ],
  "Resource": "arn:aws:kinesis:REGION:ACCOUNT_ID:stream/STREAM_NAME"
}

Configuration

FieldTypeRequiredDescription
AWS RegionSelectYesThe AWS region where the Kinesis stream is located. Options include US East (N. Virginia, Ohio), US West (N. California, Oregon), EU (Ireland, Frankfurt, London), and Asia Pacific (Tokyo, Singapore, Sydney)

Target Settings

FieldTypeRequiredDescription
Stream NameTextYesThe name of the Kinesis data stream to write to (e.g., my-kinesis-stream)
Partition Key FieldTextNoRow field to use as the partition key. Defaults to the primary key if not specified

Supported Operations

Sync Modes: Insert

Audience Sync Modes: Add

Features

  • Field Mapping: No — Kinesis publishes the full record as a JSON message
  • Schema Introspection: No — Kinesis streams are schemaless

How It Works

SignalSmith sends each row as a JSON record to the configured Kinesis data stream:

  1. Each row is serialized as a JSON object
  2. The partition key is set from the configured field (or the primary key)
  3. Records are sent to Kinesis using the PutRecords API for batch efficiency
  4. Kinesis distributes records across shards based on the partition key hash

The partition key determines which shard receives each record. Records with the same partition key are always sent to the same shard, preserving ordering per key.

Troubleshooting

Access denied

Verify the IAM user has kinesis:PutRecord and kinesis:PutRecords permissions on the target stream. Check the IAM policy resource ARN matches the stream.

Stream not found

Ensure the stream name is correct and the stream is in the selected AWS region. Stream names are case-sensitive.

Throughput exceeded

Each Kinesis shard supports up to 1,000 records/second and 1 MB/second for writes. If you see ProvisionedThroughputExceededException, increase the number of shards or reduce sync frequency.

Region mismatch

The selected region must match the region where the Kinesis stream was created. Verify the stream region in the Kinesis console.

KafkaGoogle Pub/Sub