Notes › System Design
Caching, CDN and Databases at Scale
Caching
A cache stores the results of expensive work (DB queries, API calls, rendered pages) in fast memory so the next request is served instantly.
- A DB query might take 20 ms; a Redis read takes ~0.5 ms.
- Works because traffic is skewed: a small % of data (popular products, trending posts) gets most reads (the 80/20 rule).
Where caches live
| Layer | Example |
|---|---|
| Browser | HTTP cache (Cache-Control), localStorage |
| CDN | edge servers cache images, JS, CSS, even pages |
| Load balancer / reverse proxy | Nginx, Varnish |
| Application | in-memory map / LRU inside the process |
| Distributed cache | Redis, Memcached — shared by all servers |
| Database | buffer pool, query cache |
Cache hit ratio = hits / total requests. Aim high (90%+) for hot data.
Caching strategies
Cache-aside (lazy loading) — most common
read: check cache → hit? return
→ miss? read DB → put in cache (with TTL) → return
write: update DB → delete the cache key
async function getUser(id) {
const key = `user:${id}`;
const cached = await redis.get(key);
if (cached) return JSON.parse(cached); // hit
const user = await User.findById(id).lean(); // miss → DB
if (user) await redis.set(key, JSON.stringify(user), { EX: 300 }); // 5-minute TTL
return user;
}
async function updateUser(id, data) {
await User.updateOne({ _id: id }, data);
await redis.del(`user:${id}`); // invalidate
}
Only requested data is cached; a cache failure isn't fatal. First read is slow.
The others
| Strategy | How it writes | Pros | Cons |
|---|---|---|---|
| Read-through | cache itself loads from DB on a miss | simpler app code | cache library must know the DB |
| Write-through | write to cache and DB together | cache always fresh | slower writes, caches unread data |
| Write-back (write-behind) | write to cache, DB updated later in batches | very fast writes | data loss if the cache dies |
| Write-around | write only to DB; cache fills on read | no cache pollution | first read after a write misses |
Eviction policies (cache is full — what to remove?)
| Policy | Removes | Good for |
|---|---|---|
| LRU (least recently used) | item not used for the longest time | most workloads (default choice) |
| LFU (least frequently used) | item used the fewest times | stable popularity |
| FIFO | oldest inserted | simple |
| TTL | items after a fixed time | data that goes stale |
LRU = hash map + doubly linked list → O(1) get and put (a classic coding question).
Cache problems
- Stale data / invalidation: "There are only two hard things in CS: cache invalidation and naming things." Use TTLs plus delete-on-write.
- Cache stampede (thundering herd): a hot key expires and 10,000 requests hit the DB at once. Fix: a lock so only one request rebuilds, early refresh, random TTL jitter.
- Cache penetration: requests for keys that don't exist always miss (often attacks). Fix: cache "null" results briefly, or a Bloom filter.
- Cache avalanche: many keys expire at the same time. Fix: randomise TTLs.
- Hot key: one key gets huge traffic — replicate it or add a local in-process cache.
Redis — more than a cache
Strings, hashes, lists, sets, sorted sets (leaderboards), pub/sub, streams, TTLs, atomic INCR (rate limiting, counters), distributed locks. Data lives in RAM with optional persistence (RDB snapshots / AOF log).
CDN — Content Delivery Network
A network of edge servers around the world that cache your static content close to users.
User in Mumbai → CDN edge in Mumbai (hit: ~10 ms)
↓ miss
origin server in US (~250 ms), then cached at the edge
- Benefits: lower latency, less load and bandwidth on your origin, DDoS protection, TLS at the edge.
- Pull CDN: the edge fetches from origin on the first miss (most common — Cloudflare, CloudFront).
- Push CDN: you upload content to the CDN in advance (large, rarely changing files).
- Control with headers:
Cache-Control: public, max-age=31536000, immutablefor fingerprinted files likeapp.3f9a1c.js; short orno-cachefor HTML. - Invalidate with versioned filenames (best) or a purge API.
Databases at scale
Replication
Copies of the same data on several servers.
- Primary–replica (master–slave): writes go to the primary, reads spread across replicas. Great for read-heavy apps.
- Replication lag: replicas may be slightly behind → a user might not see their own update. Fix: read-your-writes (read from the primary right after writing).
- Failover: a replica is promoted if the primary dies.
- Multi-primary: writes anywhere, but conflicts must be resolved.
- Sync replication = safe but slower; async = fast but can lose the latest writes on failure.
Partitioning and sharding
- Vertical partitioning: split by columns/features (users DB, orders DB).
- Horizontal partitioning / sharding: split rows across machines by a shard key.
| Sharding method | How | Watch out |
|---|---|---|
| Range | user_id 1–1M → shard 1 … | hot spots (new users all on the last shard) |
| Hash | hash(user_id) % N | adding shards moves most data → use consistent hashing |
| Directory | lookup table key → shard | the directory is a dependency |
Problems: cross-shard joins and transactions, rebalancing, choosing a good key (high cardinality, even spread, matches query patterns). Details in Database Sharding and Replication.
Other tools
- Indexes first — the cheapest win (Indexing, B+ Trees and SQL vs NoSQL).
- Connection pooling: reuse DB connections instead of opening one per request.
- Denormalization / materialized views for heavy reads.
- CQRS: separate models (even databases) for writes and reads.
- Archive cold data to cheaper storage.
Interview one-liners
- Cache-aside: read cache → miss → DB → fill cache; on write, update DB then delete the key.
- LRU = hash map + doubly linked list.
- CDN caches static content at edges near users; version filenames to bust caches.
- Replication scales reads and adds availability; sharding scales writes and storage.
- Use consistent hashing so adding a shard moves only a fraction of keys.
- Related: System Design Basics: Scalability, CAP Theorem and Consistency.