Getting everything ready for you…

Loading courses, problems and your progress. This takes just a moment.

0s

Projects › Advanced

Distributed Job Queue & Rate Limiter

Build your own background job queue with retries and a Redis-based rate limiter — the internals of tools every backend uses.

Level: Advanced · Time: 2 weeks · Tech stack: Node.js, Redis, Docker

Features to build

What you will learn

Step-by-step guide

1. Queue in Redis

Ready jobs in a list (or sorted set for priority/delay), job data in a hash, in-progress jobs in another list.

2. Take a job atomically

LMOVE ready processing RIGHT LEFT moves a job in one step, so two workers never get the same job.

3. Crashes and retries

Each worker sends a heartbeat. A "reaper" moves jobs stuck in processing back to ready. Failed jobs retry after 1 s, 2 s, 4 s… and land in a dead-letter queue after 5 tries.

4. Idempotent jobs

At-least-once delivery means a job can run twice — design handlers so that's safe (e.g. "send email unless already sent for this id").

5. Sliding-window rate limiter

-- KEYS[1] = key, ARGV = now, window(ms), limit
redis.call('ZREMRANGEBYSCORE', KEYS[1], 0, ARGV[1] - ARGV[2])
if redis.call('ZCARD', KEYS[1]) >= tonumber(ARGV[3]) then return 0 end
redis.call('ZADD', KEYS[1], ARGV[1], ARGV[1] .. math.random())
redis.call('PEXPIRE', KEYS[1], ARGV[2])
return 1

Running it as a Lua script makes the check-and-add atomic across servers.

6. Prove it works

Load-test with k6 or autocannon: 3 workers, 10,000 jobs, kill a worker mid-way and show no job is lost.

Level it up

On your resume

Learn on Codify