Azure and Amazon Data Stream Analytics and Processing: Amazon Kinesis, Azure Stream Analytics, and Azure Event Hub

5 min readDec 27, 2020

6 min to read

Data Stream Analytics also called event stream processing or real-time analytics is the processing and analysis of real-time data. These streaming data could be transaction data from an e-commerce website, financial trading floors, telemetry from IoT devices, and social media data.

This article gives a brief description and use cases of the data stream analytics services in AWS and Azure. It also contains resources about their prices, components, hands-on examples, code samples, and comparison.

Use Cases for Steam Analytics

Anomaly detection in credit card transactions
Anomaly detection (fraud/outliers)
Application logging
Analytics pipelines, such as clickstreams
Live dashboarding
Archiving data
Transaction processing
User telemetry processing
Device telemetry streaming
Alert system
Click steam Analytics
Analyze streaming social media data

Amazon Kinesis

Amazon Kinesis is an Amazon Web Service for easy and scalable collection, processing, and analysis of video and data streams in real-time. It can also be used for batch processing. It is can be used for data ingestion for machine learning, analytics, and other applications.

Other Amazon Web Services used with Amazon Kinesis: Kinesis Video Streams, Kinesis Data Streams, Kinesis Data Firehose, Kinesis Data Analytics, Amazon EC2, Amazon S3, Amazon Redshift, DynamoDB, Amazon EMR, Amazon Elasticsearch Service, and Amazon Lambda.

Types of Amazon Kinesis

Kinesis Video Streams — capture, process, and store video streams.

Kinesis Data Streams- capture, process, and store data streams.

Kinesis Data Firehose — capture, transform, and load data streams into AWS data stores.

Kinesis Data Analytics — process data streams in real-time with SQL or Apache Flink.

Pricing

Amazon Kinesis Data Streams Pricing is based on Shared Hour and PUT Payload Unit.

Amazon Kinesis Video Streams Pricing is based on the volume of data you ingest, store, and consume in your video streams.

Amazon Kinesis Data Firehose Pricing is based on the volume of data ingested into Kinesis Data Firehose.

Amazon Kinesis Data Analytics Pricing is based on the average number of Kinesis Processing Units (or KPUs) used to run your stream processing application which is charged per hour.

Hand-On Tutorials

Building a log Analytics Solution

Set up a kinesis Agent
Create an end to end data delivery stream using kinesis Data Firehose
Process incoming data using SQL queries using Amazon Kinesis Data Analytics
Load processed data from Kinesis Data Analytics to Amazon Elasticsearch Service
Analyze and visualize the processed data using Kibana.

Process Real-Time Stock Data Using KPL and KCL

Real-Time Hotspot Detection in Amazon Kinesis Analytics

Code Examples

Generating streams of weather data Kinesis Data Generator

Official Resources

Amazon Kinesis Amazon Kinesis Blog

Other Resources

High Performance Data Streaming with Amazon Kinesis: Best Practices and Common Pitfalls(video)

AWS Streaming Data Solution for Amazon Kinesis(article)

Streaming Data Solutions on AWS with Amazon Kinesis(pdf)

AWS Kinesis Tutorial for Beginners | Introduction to Amazon Kinesis AWS Training | Edureka(video)

Azure Event Hub

Event Hub is a real time ingestion service. The streamed data can be stored in Blob storage, Data Lake, and other storage. It can be connected to Azure Stream Analytics to analyze the data. Data can be sent to Event Hub, analysed with Azure Stream Analytics at real time, then visualized with Power BI.

Other Azure services commonly used with Azure Event Hub :- Azure Stream Analytics, Power BI, Azure blob storage, Azure event grid, Azure Function, Azure Synapse Analytics, and Azure Event Gird.

Pricing

Azure Event Hub Pricing is based on throughput unit, ingress events, and capture.

Hands-On Tutorials

Migrate captured Event Hubs data to Azure Synapse Analytics using Event Grid and Azure Functions

Data sent to an Azure event hub is captured in Azure blob storage.
When the data capture is complete, an event is generated and sent to an Azure event grid.
The event grid forwards this event data to Azure function app.
The function app uses the blob URL in the event data to retrieve the blob from the storage.
The function app migrates the blob data to Azure Synapse Analytics.

Visualize data anomalies in real-time events sent to Azure Event Hubs Stream Twitter data into Azure Databricks using Event Hubs

Code Samples

Azure Event Hubs samples

Official Resources

Azure Event Hubs documentation

Other Resources

Azure Event Hub Tutorial | Big data message streaming service(video)

Azure Event Hubs for Apache Kafka | Azure Friday(video)

How to perform data ingestion with Azure Event Hubs | Azure Makers Series(video)

Messaging with Azure Event Hubs(video)

Azure Stream Analytics

Azure Stream Analytics is a serverless real time analytics service. It uses SQL which can be extensible with custom code and built in machine learning capabilities.

Other Azure services commonly used with Azure Stream Analytics:- Azure Event Hub, Power BI, Azure blob storage, Azure event grid, Azure Function, Azure Synapse Analytics

Pricing

Azure Stream Analytics Pricing is based on the number of streaming Units Provisioned. A Streaming Unit represents the amount of memory and compute allocated to your resources.