Notes › System Design
Message Queues and Event-Driven Systems
A message queue lets one service (the producer) hand work to another (the consumer) without waiting for it. Messages wait in a broker until a consumer processes them.
Producer ──► [ msg | msg | msg ] queue/broker ──► Consumer(s)
Why use a queue
Without a queue — a signup request does everything before replying:
POST /signup → save user → send welcome email (2 s) → resize avatar (1 s) → reply = slow, and fails if email is down
With a queue:
POST /signup → save user → publish "user.created" → reply in 50 ms
├──► email worker sends the email
└──► image worker resizes the avatar
| Benefit | Meaning |
|---|---|
| Decoupling | producer doesn't know or care who consumes |
| Faster responses | slow work happens in the background |
| Load levelling | a traffic spike fills the queue; workers drain it at their pace |
| Reliability | if a consumer crashes, messages wait and are retried |
| Scalability | add more consumers to process faster |
Two patterns
| Point-to-point (work queue) | Publish/subscribe |
|---|---|
| each message is processed by one consumer | each message goes to every subscriber group |
| jobs: send email, generate PDF, encode video | events: "order placed" → billing, inventory, analytics, email all react |
| RabbitMQ queue, AWS SQS, BullMQ | Kafka topics, RabbitMQ fanout exchange, Google Pub/Sub, AWS SNS |
Kafka vs RabbitMQ
| RabbitMQ | Apache Kafka | |
|---|---|---|
| Model | message broker; queues + exchanges | distributed append-only log split into partitions |
| After consuming | message is deleted (acked) | message stays for the retention period; consumers track an offset |
| Replay old messages | ❌ | ✅ |
| Ordering | per queue | per partition |
| Throughput | high (tens of thousands/s) | very high (millions/s) |
| Routing | rich (direct, topic, fanout, headers) | by topic + partition key |
| Best for | task queues, complex routing, RPC | event streaming, logs, analytics, event sourcing |
| Others: AWS SQS (managed queue), Redis Streams / BullMQ (simple Node jobs), NATS, Pulsar. |
Kafka vocabulary
- Topic — a named stream (
orders). Partition — an ordered log inside a topic; partitions spread load. - Key — messages with the same key (e.g.
orderId) go to the same partition → kept in order. - Consumer group — consumers sharing the work; each partition is read by one consumer in the group. So max useful consumers = number of partitions.
- Offset — position in a partition; committed after processing.
Delivery guarantees
| Guarantee | Meaning | Risk |
|---|---|---|
| At most once | ack before processing | a crash loses the message |
| At least once (most common) | ack after processing | a crash after processing → duplicate |
| Exactly once | each effect happens once | hard; Kafka supports it within Kafka using transactions + idempotent producers |
Idempotent consumers — handle duplicates
Assume every message may arrive twice. Make processing safe to repeat:
async function handlePayment(msg) {
const { paymentId, orderId } = msg;
// unique index on paymentId → a second insert fails, so we skip
const already = await Processed.findOne({ paymentId });
if (already) return; // duplicate — ignore
await Order.updateOne({ _id: orderId }, { $set: { status: "paid" } });
await Processed.create({ paymentId });
}
Better still: do the check and the update in one transaction, or rely on a unique key.
Failures and retries
- Retry with exponential backoff (1 s, 2 s, 4 s, 8 s…) for temporary errors.
- Dead-letter queue (DLQ): after N failed attempts, move the message aside for inspection — one bad "poison" message must not block the queue.
- Visibility timeout (SQS): while a consumer works on a message it is hidden; if not deleted in time it reappears.
The dual-write problem and the outbox pattern
Saving to the DB and then publishing to the queue can half-fail (DB saved, publish lost).
Transactional outbox: in the same DB transaction insert the business row and an outbox row; a separate relay reads the outbox and publishes, marking rows as sent.
Ordering and back-pressure
- Global ordering doesn't scale; order per key (partition by
userId/orderId). - If producers are faster than consumers, the queue grows → autoscale consumers on queue length / consumer lag, alert on it, rate-limit producers.
Event-driven architecture patterns
- Event notification: "order 42 placed" — consumers fetch details if needed.
- Event-carried state transfer: the event includes the data consumers need.
- Event sourcing: store every change as an event; current state = replay of events.
- Saga: a multi-service transaction as a chain of local steps with compensating actions (payment failed → cancel the reserved stock).
- CQRS: separate write and read models, kept in sync through events.
Simple Node example (BullMQ + Redis)
import { Queue, Worker } from "bullmq";
const connection = { host: "localhost", port: 6379 };
export const emailQueue = new Queue("emails", { connection });
await emailQueue.add("welcome", { userId }, { attempts: 5, backoff: { type: "exponential", delay: 1000 } });
new Worker("emails", async (job) => {
await sendWelcomeEmail(job.data.userId); // throw → retried
}, { connection, concurrency: 10 });
When not to use a queue
- The user needs the result right now (login, search) → a direct call.
- Tiny apps — a queue adds infrastructure, monitoring and eventual consistency.
Interview one-liners
- Queues decouple services, smooth spikes and allow retries.
- Work queue: one consumer per message; pub/sub: every subscriber gets it.
- Kafka keeps messages (replay, offsets); RabbitMQ deletes after ack.
- At-least-once delivery → make consumers idempotent.
- Use a DLQ for poison messages; outbox pattern for reliable publishing.
- Related: Design a Chat Application, Load Balancing.