Batch vs Streaming in Data Engineering on AWS

Introduction

In the field of data engineering, understanding the differences between batch and streaming data processing is crucial. This lesson explores these two paradigms, their use cases, and how they can be implemented using AWS services.

Definitions

Batch Processing

Batch processing is suitable for scenarios where data can be processed periodically. Common characteristics include:

• Data is accumulated over time.
• Processing occurs at scheduled intervals.
• Latency is acceptable, often on the order of hours or days.

Example in AWS

A common service for batch processing in AWS is AWS Glue. Below is an example of how to create a Glue job:

import boto3

glue = boto3.client('glue')

response = glue.create_job(
    Name='BatchJob',
    Role='AWSGlueServiceRole',
    Command={
        'Name': 'glueetl',
        'ScriptLocation': 's3://my-bucket/scripts/my_script.py',
        'PythonVersion': '3'
    },
    MaxRetries=0,
    Timeout=60
)

print(response)
                

Streaming Processing

Streaming processing is ideal for cases where immediate processing of data is required. Characteristics include:

• Data is processed in real-time.
• Lower latency, often in milliseconds.
• Continuous data flow.

Example in AWS

A popular service for streaming data processing on AWS is Amazon Kinesis. Below is an example of how to put records into a Kinesis stream:

import boto3
import json

kinesis = boto3.client('kinesis')

data = {
    'message': 'Hello, World!'
}

response = kinesis.put_record(
    StreamName='MyStream',
    Data=json.dumps(data),
    PartitionKey='partitionkey1'
)

print(response)
                

Best Practices

For Batch Processing

• Optimize data storage formats such as Parquet or ORC.
• Use partitioning to improve read performance.
• Schedule jobs during off-peak hours to reduce resource contention.

For Streaming Processing

• Implement backpressure handling to manage data spikes.
• Use proper monitoring tools to track stream health.
• Utilize windowing to aggregate data over specific time intervals.

FAQ