System Design FAQ: Top Questions

An IoT Data Collection Platform ingests telemetry data from millions of distributed sensors and devices, processes the data in near real-time, stores it efficiently, and provides insights or control commands back to devices.

📋 Functional Requirements

• Ingest data from millions of IoT devices (temperature, GPS, status)
• Support multiple protocols (MQTT, HTTP, CoAP)
• Store raw and transformed data for real-time and historical analysis
• Push commands to devices (bidirectional communication)

📦 Non-Functional Requirements

• High throughput and fault tolerance
• Secure authentication and encrypted transport
• Horizontal scalability and multi-region support

🏗️ Architecture Overview

• Device Gateway: MQTT broker or HTTP API to ingest messages
• Stream Processor: Apache Flink, Kafka Streams, or Spark
• Storage: Raw data to S3, transformed data to InfluxDB/TSDB
• Control Plane: Push down configuration or actuation commands

📡 MQTT Message Format

{
  "device_id": "sensor-4492",
  "timestamp": "2025-06-11T15:00:00Z",
  "metrics": {
    "temperature": 72.4,
    "humidity": 40.5
  }
}
        

⚙️ EMQX MQTT Broker Config

listener.tcp.external = 0.0.0.0:1883
auth.user.password_hash = sha256
mqtt.max_packet_size = 1MB
mqtt.keepalive = 60
        

🔐 Security Measures

• TLS for transport encryption (port 8883 for MQTT over TLS)
• Device identity via X.509 or JWT token provisioning
• Rate limiting and message validation

🔄 Real-Time Aggregation (Flink SQL)

SELECT
  device_id,
  TUMBLE_START(ts, INTERVAL '1 MINUTE') as minute,
  AVG(metrics.temperature) as avg_temp
FROM iot_stream
GROUP BY device_id, TUMBLE(ts, INTERVAL '1 MINUTE');
        

📥 Storage Backends

• Hot storage: InfluxDB, TimescaleDB
• Cold storage: S3, BigQuery, Athena for querying
• Message buffer: Kafka or AWS IoT Core Rules

📈 Dashboarding and Alerts

• Grafana for charts, streaming panels
• Alertmanager or AWS CloudWatch alarms on thresholds

🧰 Tools & Infra

• Broker: EMQX, Mosquitto, AWS IoT Core
• Processing: Kafka Streams, Flink, AWS Lambda
• Database: InfluxDB, PostgreSQL + Timescale

📌 Final Insight

IoT platforms require efficient ingestion and storage systems that can handle bursty traffic and scale globally. MQTT is the de facto protocol for device-to-cloud, and stream processing ensures real-time insights and feedback control.