Getting everything ready for you…

Loading courses, problems and your progress. This takes just a moment.

0s

Notes › System Design

Message Queues and Event-Driven Systems

A message queue lets one service (the producer) hand work to another (the consumer) without waiting for it. Messages wait in a broker until a consumer processes them.

Producer ──► [ msg | msg | msg ]  queue/broker ──► Consumer(s)

Why use a queue

Without a queue — a signup request does everything before replying:

POST /signup → save user → send welcome email (2 s) → resize avatar (1 s) → reply   = slow, and fails if email is down

With a queue:

POST /signup → save user → publish "user.created" → reply in 50 ms
                                   ├──► email worker sends the email
                                   └──► image worker resizes the avatar
Benefit Meaning
Decoupling producer doesn't know or care who consumes
Faster responses slow work happens in the background
Load levelling a traffic spike fills the queue; workers drain it at their pace
Reliability if a consumer crashes, messages wait and are retried
Scalability add more consumers to process faster

Two patterns

Point-to-point (work queue) Publish/subscribe
each message is processed by one consumer each message goes to every subscriber group
jobs: send email, generate PDF, encode video events: "order placed" → billing, inventory, analytics, email all react
RabbitMQ queue, AWS SQS, BullMQ Kafka topics, RabbitMQ fanout exchange, Google Pub/Sub, AWS SNS

Kafka vs RabbitMQ

RabbitMQ Apache Kafka
Model message broker; queues + exchanges distributed append-only log split into partitions
After consuming message is deleted (acked) message stays for the retention period; consumers track an offset
Replay old messages ❌ ✅
Ordering per queue per partition
Throughput high (tens of thousands/s) very high (millions/s)
Routing rich (direct, topic, fanout, headers) by topic + partition key
Best for task queues, complex routing, RPC event streaming, logs, analytics, event sourcing
Others: AWS SQS (managed queue), Redis Streams / BullMQ (simple Node jobs), NATS, Pulsar.

Kafka vocabulary

Delivery guarantees

Guarantee Meaning Risk
At most once ack before processing a crash loses the message
At least once (most common) ack after processing a crash after processing → duplicate
Exactly once each effect happens once hard; Kafka supports it within Kafka using transactions + idempotent producers

Idempotent consumers — handle duplicates

Assume every message may arrive twice. Make processing safe to repeat:

async function handlePayment(msg) {
  const { paymentId, orderId } = msg;
  // unique index on paymentId → a second insert fails, so we skip
  const already = await Processed.findOne({ paymentId });
  if (already) return;                      // duplicate — ignore
  await Order.updateOne({ _id: orderId }, { $set: { status: "paid" } });
  await Processed.create({ paymentId });
}

Better still: do the check and the update in one transaction, or rely on a unique key.

Failures and retries

The dual-write problem and the outbox pattern

Saving to the DB and then publishing to the queue can half-fail (DB saved, publish lost). Transactional outbox: in the same DB transaction insert the business row and an outbox row; a separate relay reads the outbox and publishes, marking rows as sent.

Ordering and back-pressure

Event-driven architecture patterns

Simple Node example (BullMQ + Redis)

import { Queue, Worker } from "bullmq";
const connection = { host: "localhost", port: 6379 };

export const emailQueue = new Queue("emails", { connection });
await emailQueue.add("welcome", { userId }, { attempts: 5, backoff: { type: "exponential", delay: 1000 } });

new Worker("emails", async (job) => {
  await sendWelcomeEmail(job.data.userId);   // throw → retried
}, { connection, concurrency: 10 });

When not to use a queue

Interview one-liners

More System Design notes