Amazon Kinesis

Amazon Kinesis is a suite of services on AWS designed to make working with massive amounts of real-time streaming data easier. It provides solutions for the entire data journey – collecting, buffering, processing, and analyzing real-time data at scale.

Key Offerings

Kinesis Data Streams (KDS): The foundation of the Kinesis family. KDS lets you build custom applications to capture and process streaming data. Some important capabilities are:
- Continuous Capture: Streams can receive data from thousands of sources simultaneously (website clickstreams, IoT device telemetry, logs, etc.).
- Data Retention: Data remains accessible for up to 7 days (extendable to 365 days), for revisiting and replaying if needed.
- Real-Time Processing: Kinesis libraries support building custom apps to analyze data on the fly as it arrives.
Kinesis Data Firehose (KDF): Offers the easiest way to deliver streaming data to destinations. Think of it as a pre-built delivery pipeline focusing on:
- No-Code Setup: Just configure a data source and a destination (like an S3 bucket, data warehouse, or analytics services), and KDF handles the rest.
- Transformations: Option for basic data transformation (format conversion, simple compression) while in transit.
- Reliability: Automatic scaling and management of the delivery process.
Kinesis Data Analytics (KDA): SQL-based or Java/Scala-based analysis and querying directly on top of streaming data within Kinesis Data Streams or Kinesis Data Firehose.
- Real-Time Analysis: Generate insights from your data streams with SQL-like syntax to run continuous queries.
- Flexible Deployment: KDA applications can be built with Kinesis' managed Flink environment or deploy your own containerized Flink applications.

Strengths & Weaknesses with Use Cases

Offering	Strengths	Weaknesses	Use Cases
Kinesis Data Streams	Flexible, highly customizable, low-latency	Requires programming effort, setup overhead	IoT device telemetry, financial trading, log aggregation
Kinesis Data Firehose	Serverless, easy setup, transformations	Less control over processing details	Delivering data to S3, loading Redshift
Kinesis Data Analytics	SQL or Flink for easy analysis, flexibility	Analysis limitations compared to dedicated tools	Website activity monitoring, live anomaly detection

Important Notes

Beyond Basics: Consider Kinesis Video Streams for real-time video processing and Amazon Managed Streaming for Kafka (MSK) for integration with Apache Kafka.
Choosing Wisely: Factor in the technical skills of your team, complexity of needs, and existing infrastructure when selecting the right Kinesis offering.

Data Shards

Data Firehose