Getting everything ready for you…

Loading courses, problems and your progress. This takes just a moment.

0s

Notes › System Design

Load Balancing

When one server isn't enough, you run several copies and put a load balancer (LB) in front. It spreads incoming requests across the servers so none is overloaded, and stops sending traffic to servers that are down. (Start with System Design Basics: Scalability.)

                 ┌──► App server 1
Users ──► LB ────┼──► App server 2
                 └──► App server 3

What a load balancer gives you

Where load balancers sit

Layer 4 vs Layer 7

L4 (transport) L7 (application)
Looks at IP + port (TCP/UDP) HTTP: URL, headers, cookies
Can route /api and /images differently ❌ ✅
Speed very fast a bit slower (parses HTTP)
TLS passes through usually terminates
Examples AWS NLB, HAProxy (TCP mode) Nginx, AWS ALB, Envoy, Traefik

Algorithms

Algorithm How Good when
Round robin 1, 2, 3, 1, 2, 3… servers are equal, requests are similar
Weighted round robin bigger servers get more turns servers differ in power
Least connections send to the server with the fewest active connections requests take very different times (uploads, WebSockets)
Least response time fastest + fewest connections latency matters
IP hash hash(client IP) % N same client → same server
Random / power of two choices pick 2 random servers, use the less loaded huge, simple systems
Consistent hashing servers and keys on a hash ring caches/shards where adding a server must move few keys

Why plain hash % N hurts, and consistent hashing

With hash(key) % N, changing N from 4 to 5 remaps ~80% of keys — every cache misses at once. With consistent hashing, servers and keys sit on a ring (0…2³²); each key goes to the next server clockwise. Adding or removing a server moves only ~1/N of the keys. Virtual nodes (each server placed many times on the ring) keep the load even. Used by DynamoDB, Cassandra, CDNs and distributed caches.

Health checks

The LB calls something like GET /health every few seconds.

Sticky sessions (session affinity)

The LB sends a user to the same server every time (cookie or IP hash) — needed if the server keeps session data in its memory.

Avoiding a single point of failure

The LB itself can fail. Solutions:

Nginx as a load balancer

upstream app {
    least_conn;
    server 10.0.0.11:5000 weight=2;
    server 10.0.0.12:5000;
    server 10.0.0.13:5000 backup;      # used only if the others are down
}
server {
    listen 443 ssl;
    server_name codify.example;
    location / {
        proxy_pass http://app;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

Behind a proxy, read the real client IP from X-Forwarded-For (in Express: app.set("trust proxy", 1)).

Load balancer vs reverse proxy vs API gateway

Main job
Reverse proxy one front door for servers (TLS, caching, compression)
Load balancer spread traffic across many identical servers
API gateway routing to many different services + auth, rate limits, API keys
Nginx can do all three; the names describe the role.

Back-of-envelope

10 million requests/day ≈ 10⁷ / 86,400 ≈ 115 requests/second average; peak ≈ 3–5× → ~500 rps. If one server handles 200 rps → 3 servers + 1 spare (N+1).

Interview one-liners

More System Design notes