Notes › System Design
System Design Basics: Scalability
Scalability = the system keeps performing well as load grows (more users, more data, more requests).
Vertical vs horizontal scaling
| Vertical (scale up) | Horizontal (scale out) | |
|---|---|---|
| How | bigger machine — more CPU, RAM, SSD | more machines behind a load balancer |
| Limit | hardware ceiling | almost unlimited |
| Failure | single point of failure | one node dies, others serve |
| Complexity | simple, no code changes | needs stateless design, load balancing, data distribution |
| Cost | gets expensive fast | commodity machines |
| Example | upgrading the DB server from 16 GB to 128 GB | 2 → 20 Node.js servers |
Real systems start vertical (cheap and simple) and go horizontal when needed.
Stateless servers
To scale out, any request must be able to go to any server. So don't keep user state (sessions, uploaded files) in server memory or local disk:
- Sessions → JWT or a shared Redis store.
- Files → object storage (S3).
- Now servers are interchangeable — add or remove them freely (auto-scaling).
Key metrics
| Metric | Meaning |
|---|---|
| Latency | time for one request (ms). Track p50, p95, p99, not just the average |
| Throughput | requests handled per second (RPS/QPS) |
| Availability | % of time the system works |
| Durability | data is never lost |
| Consistency | every read sees the latest write |
The "nines" of availability
| Availability | Downtime per year |
|---|---|
| 99% | 3.65 days |
| 99.9% | 8.76 hours |
| 99.99% | 52.6 minutes |
| 99.999% | 5.26 minutes |
Two components in series (both needed): 99.9% × 99.9% = 99.8%. Two in parallel (either works): 1 − 0.001² = 99.9999%. Redundancy raises availability.
Latency numbers to know (roughly)
| Operation | Time |
|---|---|
| L1 cache / RAM read | ~1 ns / ~100 ns |
| Read 1 MB from RAM | ~10 µs |
| SSD random read | ~100 µs |
| Round trip in the same data centre | ~0.5 ms |
| Read 1 MB from SSD | ~1 ms |
| Disk (HDD) seek | ~10 ms |
| Round trip India ↔ US | ~150–250 ms |
Memory ≫ SSD ≫ network ≫ disk — hence caching.
Growing an app: 1 user → millions
- Single server: app + DB on one machine.
- Separate the database onto its own server — scale them independently.
- Load balancer + multiple app servers (stateless) — no single point of failure for the app tier. See Load Balancing.
- Database replication: one primary for writes, read replicas for reads (most apps are read-heavy).
- Cache (Redis) for hot data and a CDN for static files — see Caching, CDN and Databases at Scale.
- Message queue for slow work (emails, video processing) — the request returns immediately; workers process later. See Message Queues.
- Sharding the database when one primary can't handle writes or storage — see Sharding and Replication.
- Multiple data centres / regions, geo-DNS routing users to the nearest one.
- Split into services (microservices) owned by different teams, plus logging, metrics and alerting.
Users → DNS → CDN (static)
↓
Load Balancer
↙ ↓ ↘
App1 App2 App3 (stateless, auto-scaled)
↘ ↓ ↙
Redis cache Queue → Workers
↓
DB primary → read replicas (→ shards)
Single point of failure (SPOF)
Any component whose failure brings everything down. Remove SPOFs with redundancy (2+ load balancers, DB failover, multiple zones) and health checks.
Back-of-the-envelope estimation
Useful shortcuts: 1 day ≈ 86,400 s ≈ 10⁵ s; 1 million requests/day ≈ 12 RPS. Example — a photo app with 10 M daily active users:
- Each user views 50 photos/day → 500 M reads/day ≈ 500M / 10⁵ ≈ 5,000 reads/s (peak ~2–3× = 15,000).
- 10% of users upload 1 photo/day → 1 M uploads/day ≈ 12 writes/s.
- Storage: 1 M × 2 MB = 2 TB/day → ~730 TB/year → object storage + CDN, not the DB.
- Read:write ≈ 400:1 → read-heavy → cache and replicas.
Answering a system design interview
- Clarify requirements (5 min): functional (what it does) and non-functional (scale, latency, availability, consistency).
- Estimate scale: users, QPS, storage, bandwidth.
- API design: main endpoints.
- Data model: entities, SQL vs NoSQL.
- High-level design: boxes and arrows — clients, LB, services, DB, cache, queue.
- Deep dive into 1–2 hard parts (feed generation, ID generation, hot keys).
- Bottlenecks and trade-offs: SPOFs, scaling each tier, failure handling, monitoring. Say trade-offs out loud — there's no single correct answer.
Interview one-liners
- Scale up = bigger machine; scale out = more machines (needs stateless servers).
- Measure p99 latency, not just the average.
- Redundancy removes single points of failure and raises availability.
- Read-heavy → cache + replicas; write-heavy → sharding + queues.
- Practise: Design a URL Shortener, Design a Chat Application.