Notes › System Design
Load Balancing
When one server isn't enough, you run several copies and put a load balancer (LB) in front. It spreads incoming requests across the servers so none is overloaded, and stops sending traffic to servers that are down. (Start with System Design Basics: Scalability.)
┌──► App server 1
Users ──► LB ────┼──► App server 2
└──► App server 3
What a load balancer gives you
- Scalability: add servers to handle more traffic (horizontal scaling).
- Availability: a crashed server is removed automatically.
- Zero-downtime deploys: drain one server, update it, put it back (rolling deploy).
- Often also: TLS termination, compression, caching, rate limiting, a single public IP.
Where load balancers sit
- Between users and web/app servers (most common).
- Between app servers and internal services or database read replicas.
- DNS / global level: GeoDNS or anycast sends users to the nearest region.
Layer 4 vs Layer 7
| L4 (transport) | L7 (application) | |
|---|---|---|
| Looks at | IP + port (TCP/UDP) | HTTP: URL, headers, cookies |
Can route /api and /images differently |
❌ | ✅ |
| Speed | very fast | a bit slower (parses HTTP) |
| TLS | passes through | usually terminates |
| Examples | AWS NLB, HAProxy (TCP mode) | Nginx, AWS ALB, Envoy, Traefik |
Algorithms
| Algorithm | How | Good when |
|---|---|---|
| Round robin | 1, 2, 3, 1, 2, 3… | servers are equal, requests are similar |
| Weighted round robin | bigger servers get more turns | servers differ in power |
| Least connections | send to the server with the fewest active connections | requests take very different times (uploads, WebSockets) |
| Least response time | fastest + fewest connections | latency matters |
| IP hash | hash(client IP) % N |
same client → same server |
| Random / power of two choices | pick 2 random servers, use the less loaded | huge, simple systems |
| Consistent hashing | servers and keys on a hash ring | caches/shards where adding a server must move few keys |
Why plain hash % N hurts, and consistent hashing
With hash(key) % N, changing N from 4 to 5 remaps ~80% of keys — every cache misses at once. With consistent hashing, servers and keys sit on a ring (0…2³²); each key goes to the next server clockwise. Adding or removing a server moves only ~1/N of the keys. Virtual nodes (each server placed many times on the ring) keep the load even. Used by DynamoDB, Cassandra, CDNs and distributed caches.
Health checks
The LB calls something like GET /health every few seconds.
- Active: periodic probes; after k failures, mark unhealthy; after k successes, add back.
- Passive: watch real traffic for errors/timeouts.
A good
/healthchecks the app is able to serve (e.g. can reach the DB), but stays cheap.
Sticky sessions (session affinity)
The LB sends a user to the same server every time (cookie or IP hash) — needed if the server keeps session data in its memory.
- ❌ Uneven load; a server crash loses its users' sessions.
- ✅ Better: make servers stateless — store sessions in Redis or use JWTs, so any server can handle any request.
Avoiding a single point of failure
The LB itself can fail. Solutions:
- Active–passive pair sharing a floating/virtual IP (keepalived, VRRP); the standby takes over in seconds.
- Active–active behind DNS or anycast.
- Managed cloud LBs (AWS ELB, GCP LB) are already redundant across zones.
Nginx as a load balancer
upstream app {
least_conn;
server 10.0.0.11:5000 weight=2;
server 10.0.0.12:5000;
server 10.0.0.13:5000 backup; # used only if the others are down
}
server {
listen 443 ssl;
server_name codify.example;
location / {
proxy_pass http://app;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
Behind a proxy, read the real client IP from X-Forwarded-For (in Express: app.set("trust proxy", 1)).
Load balancer vs reverse proxy vs API gateway
| Main job | |
|---|---|
| Reverse proxy | one front door for servers (TLS, caching, compression) |
| Load balancer | spread traffic across many identical servers |
| API gateway | routing to many different services + auth, rate limits, API keys |
| Nginx can do all three; the names describe the role. |
Back-of-envelope
10 million requests/day ≈ 10⁷ / 86,400 ≈ 115 requests/second average; peak ≈ 3–5× → ~500 rps. If one server handles 200 rps → 3 servers + 1 spare (N+1).
Interview one-liners
- L4 routes by IP/port; L7 routes by HTTP content.
- Least connections suits long or uneven requests.
- Consistent hashing moves only ~1/N keys when servers change.
- Make app servers stateless so sticky sessions aren't needed.
- The LB must be redundant too.
- Related: Caching, CDN and Databases at Scale, Network Devices and Common Ports.