System Design FAQ: Top Questions

A Messaging Queue System is used to decouple producers and consumers of data, enabling asynchronous communication, buffering, and fault tolerance in distributed systems. Examples include Kafka, RabbitMQ, and Amazon SQS.

🎯 Core Use Cases

• Asynchronous processing of user actions
• Event-driven architectures (microservices)
• Log aggregation and streaming pipelines
• Buffering for peak traffic (e.g., during flash sales)

🧱 Architecture Components

• Producer: Sends messages to the queue
• Broker: Manages queues, topics, delivery guarantees
• Consumer: Processes messages, may be parallelized
• Storage: Persist messages (disk/log-based or in-memory)

⚙️ Types of Messaging Patterns

• Point-to-Point: One consumer receives each message
• Pub/Sub: Messages are broadcasted to all subscribers

📦 Example: RabbitMQ Consumer/Producer (Python + Pika)

# producer.py
import pika

conn = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
ch = conn.channel()
ch.queue_declare(queue='task_queue')

ch.basic_publish(exchange='',
                 routing_key='task_queue',
                 body='Hello World!')
print(" [x] Sent 'Hello World!'")
conn.close()
        

# consumer.py
import pika

conn = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
ch = conn.channel()
ch.queue_declare(queue='task_queue')

def callback(ch, method, properties, body):
    print(f" [x] Received {body}")

ch.basic_consume(queue='task_queue',
                 on_message_callback=callback,
                 auto_ack=True)

print(' [*] Waiting for messages. To exit press CTRL+C')
ch.start_consuming()
        

☁️ Kafka High-Level Architecture

• Topics: Logical stream categories
• Partitions: Parallelism & ordering units
• Brokers: Store partition logs
• ZooKeeper: Manages broker coordination (Kafka ≤2.8)

🔐 Kafka Producer Properties (config)

acks=all
retries=3
compression.type=snappy
batch.size=32768
linger.ms=10
        

🚀 Delivery Semantics

• At most once: No retries, risk of loss
• At least once: Retry on failure, potential duplicates
• Exactly once: Requires idempotency and transactions (complex)

📈 Monitoring Metrics

• Lag per partition
• Consumer throughput
• Broker disk usage
• Message drop rate (alerts!)

📌 Final Insight

A message queue is a powerful way to build decoupled and resilient systems. Selecting the right architecture and delivery guarantees depends on your system’s tolerance for latency, throughput, and message loss.