System Design Mastery
1. System Design Framework
Four-Step Framework
Step 1: REQUIREMENTS ──→ Functional + Non-Functional (QPS, latency, storage, availability)
Step 2: ESTIMATION ───→ Back-of-envelope: users × requests × data size × redundancy
Step 3: HIGH-LEVEL ───→ Draw architecture with 5-7 boxes (client → LB → API → DB → cache)
Step 4: DEEP-DIVE ────→ Pick 2-3 components, discuss tradeoffs, bottlenecks, alternatives
Estimation Cheat Sheet
| Metric | Formula | Example |
|--------|---------|---------|
| QPS | DAU × requests/user / 86400 | 10M × 20 / 86400 ≈ 2,300 QPS |
| Storage | records/day × record_size × retention | 1M × 1KB × 365d = 365GB |
| Bandwidth | QPS × avg_response_size | 2300 × 50KB = 115MB/s |
| Peak QPS | avg QPS × 2-3x | 2300 × 3 = 6,900 QPS |
2. CAP Theorem & Consistency Models
CAP Theorem: Choose 2 of 3
┌─────────────────────────────────────────────────────────────┐
│ CAP Tradeoffs │
│ │
│ AP (Availability + Partition Tolerance) │
│ └─ DynamoDB, Cassandra, CouchDB — eventual consistency │
│ │
│ CP (Consistency + Partition Tolerance) │
│ └─ ZooKeeper, etcd, HBase — unavailable during partition │
│ │
│ CA (Consistency + Availability) — impossible in distributed│
│ └─ RDBMS (single-node, no partition tolerance) │
└─────────────────────────────────────────────────────────────┘
PACELC Extension
Partition → Availability vs Consistency
Else → Latency vs Consistency
DynamoDB: P→A, E→L (fast reads, eventual)
Google Spanner: P→C, E→C (strong consistency always)
3. Load Balancing
Decision Tree
Need load balancing? ─┬─ Layer 4 (TCP/UDP) ──→ HAProxy, NLB, iptables
│ Fast, no HTTP awareness
├─ Layer 7 (HTTP) ──→ Nginx, ALB, Traefik, Kong
│ Path routing, SSL termination, rate limiting
└─ Global (DNS) ──→ Cloudflare, Route 53, GeoDNS
Latency-based routing, DDoS protection
Nginx Load Balancer Config
upstream backend {
least_conn; # Algorithm: round_robin | least_conn | ip_hash | random
server 10.0.1.1:8080 weight=5;
server 10.0.1.2:8080 weight=3;
server 10.0.1.3:8080 backup; # Only if primaries fail
keepalive 32; # Connection pooling
}
server {
listen 443 ssl http2;
ssl_certificate /etc/ssl/cert.pem;
ssl_certificate_key /etc/ssl/key.pem;
location / {
proxy_pass http://backend;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $host;
# Health check (active)
proxy_next_upstream error timeout http_502 http_503;
proxy_connect_timeout 5s;
proxy_read_timeout 30s;
}
# Health check endpoint (passive)
location /health {
proxy_pass http://backend/health;
access_log off;
}
}
HAProxy TCP Load Balancer
# /etc/haproxy/haproxy.cfg
global
maxconn 100000
daemon
defaults
mode tcp
timeout connect 5s
timeout client 30s
timeout server 30s
retries 3
frontend mysql_front
bind *:3306
default_backend mysql_back
backend mysql_back
balance leastconn
option mysql-check user haproxy_check
server mysql1 10.0.1.1:3306 check inter 5s rise 3 fall 2
server mysql2 10.0.1.2:3306 check inter 5s rise 3 fall 2 backup
4. Caching
Cache Strategy Decision Tree
Need caching? ─┬─ Read-heavy, static data ──→ Cache-Aside (lazy load)
├─ Write-heavy, must be fresh ──→ Write-Through (sync write)
├─ Batch writes acceptable ──→ Write-Behind (async flush)
└─ Session/state data ──→ Distributed cache (Redis Cluster)
Redis Cache-Aside Pattern
import redis
import json
r = redis.Redis(host='redis-primary', port=6379, db=0, decode_responses=True)
async def get_user(user_id: str) -> dict:
"""Cache-aside: check cache first, miss → fetch from DB → populate cache"""
cache_key = f"user:{user_id}"
# 1. Check cache
cached = r.get(cache_key)
if cached:
return json.loads(cached)
# 2. Cache miss → fetch from DB
user = await db.fetch_one("SELECT * FROM users WHERE id = $1", user_id)
# 3. Populate cache with TTL
r.setex(cache_key, 3600, json.dumps(user)) # 1-hour TTL
return user
async def invalidate_user(user_id: str):
"""Delete cached entry on update"""
r.delete(f"user:{user_id}")
r.delete(f"user_posts:{user_id}") # Invalidate related caches
Redis Cluster Config
# redis.conf for each node (6-node cluster: 3 masters + 3 replicas)
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 15000
cluster-announce-ip 10.0.1.1
cluster-announce-port 6379
cluster-announce-bus-port 16379
Create cluster
redis-cli --cluster create \
10.0.1.1:6379 10.0.1.2:6379 10.0.1.3:6379 \
10.0.1.4:6379 10.0.1.5:6379 10.0.1.6:6379 \
--cluster-replicas 1
5. Database Sharding & Replication
Sharding Decision Tree
Data too large for one node? ─┬─ No ──→ Vertical scaling (bigger instance)
├─ Yes, <1TB ──→ Read replicas first
├─ Yes, 1-10TB ──→ Range or Hash sharding
└─ Yes, >10TB ──→ Vitess/CockroachDB (auto-sharding)
Hash-Based Sharding
def get_shard(key: str, num_shards: int = 16) -> int:
"""Consistent hash for shard routing"""
return int(hashlib.md5(key.encode()).hexdigest(), 16) % num_shards
Resharding with virtual nodes (256 vnodes per physical node)
RING = ConsistentHash(vnodes=256, replicas=3)
RING.add_node('shard-0', 'db-shard-0.internal:5432')
RING.add_node('shard-1', 'db-shard-1.internal:5432')
RING.add_node('shard-2', 'db-shard-2.internal:5432')
Only ~25% of keys move when adding a 4th node (vs 75% with naive modulo)
MySQL Master-Slave Replication
# my.cnf (master)
[mysqld]
server-id = 1
log-bin = mysql-bin
binlog-format = ROW
gtid-mode = ON
enforce-gtid-consistency = ON
binlog-do-db = app_db
sync-binlog = 1
innodb-flush-log-at-trx-commit = 1
my.cnf (replica)
[mysqld]
server-id = 2
relay-log = relay-bin
read-only = 1
gtid-mode = ON
enforce-gtid-consistency = ON
6. Message Queues & Event Streaming
Queue Selection Decision Tree
Which messaging system? ─┬─ Simple task queue, <10K msg/s ──→ Redis Streams / SQS
├─ Pub/Sub, fan-out, routing ──→ RabbitMQ
├─ High throughput, event streaming ──→ Apache Kafka
└─ Exactly-once, transactions ──→ Kafka + Transactions API
Kafka Producer (Python)
from confluent_kafka import Producer, KafkaError
conf = {
'bootstrap.servers': 'kafka-1:9092,kafka-2:9092,kafka-3:9092',
'client.id': 'order-service',
'acks': 'all', # Wait for all ISR acks
'retries': 3,
'retry.backoff.ms': 1000,
'enable.idempotence': True, # Exactly-once semantics
'compression.type': 'lz4',
'linger.ms': 50, # Batch small messages
'batch.size': 32768,
}
producer = Producer(conf)
def delivery_report(err, msg):
if err:
print(f"Delivery failed: {err}")
else:
print(f"Delivered to {msg.topic()} [{msg.partition()}] @ offset {msg.offset()}")
Produce message
producer.produce(
topic='orders',
key=str(order_id).encode(),
value=json.dumps(order).encode(),
headers={'source': b'order-service', 'trace_id': trace_id.encode()},
callback=delivery_report
)
producer.flush()
RabbitMQ with Dead Letter Queue
# Consumer config with DLQ
spring:
rabbitmq:
host: rabbitmq
listener:
simple:
acknowledge-mode: manual
prefetch: 10
retry:
max-attempts: 3
max-interval: 5s
Queue declaration with DLQ
@Queue(value = "orders", arguments = {
@Argument(name = "x-dead-letter-exchange", value = "dlq.exchange"),
@Argument(name = "x-dead-letter-routing-key", value = "orders.dlq"),
@Argument(name = "x-message-ttl", value = "86400000") # 24h TTL
})
7. API Gateway Patterns
Kong Rate Limiting Config
# Kong declarative config (kong.yml)
_format_version: "3.0"
services:
- name: order-service
url: http://order-service:8080
routes:
- name: orders
paths:
- /api/orders
methods:
- GET
- POST
plugins:
- name: rate-limiting
config:
minute: 100
hour: 5000
policy: redis
redis_host: redis
redis_port: 6379
redis_database: 1
- name: key-auth
config:
key_names:
- apikey
- name: request-transformer
config:
add:
headers:
- "X-Gateway-Timestamp:$(date)"
8. Microservices Patterns
Circuit Breaker (Python)
from circuitbreaker import circuit
@circuit(failure_threshold=5, recovery_timeout=30, expected_exception=ConnectionError)
async def call_payment_service(order_id: str):
"""Circuit breaker wraps external service calls"""
async with httpx.AsyncClient() as client:
response = await client.post(
f"http://payment-service:8080/charge",
json={"order_id": order_id},
timeout=5.0
)
response.raise_for_status()
return response.json()
Fallback when circuit is open
@circuit(fallback_function=lambda: {"status": "pending", "reason": "service_unavailable"})
async def call_payment_with_fallback(order_id: str):
return await call_payment_service(order_id)
Saga Pattern (Choreography)
Order Flow (Saga Choreography):
┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Order Svc │───→│ Payment Svc │───→│ Inventory │───→│ Shipping │
│ (created) │ │ (charged) │ │ (reserved) │ │ (scheduled) │
└──────┬───────┘ └──────┬───────┘ └──────┬───────┘ └──────────────┘
│ │ │
│ COMPENSATE │ COMPENSATE │ COMPENSATE
↓ ↓ ↓
Cancel Order Refund Payment Release Inventory
Each step publishes an event; next service subscribes and reacts.
If any step fails, compensating transactions fire in reverse order.
9. Flowcharts & Decision Diagrams
Architecture Selection Decision Tree
What are you building? ──┬─ CRUD web app <1K QPS ──→ Monolith (Rails/Django/NestJS)
├─ Real-time <10K connections ──→ WebSocket server (Socket.IO/ws)
├─ High throughput streaming ──→ Kafka + microservices
├─ Compute-heavy jobs ──→ Task queue (Celery/BullMQ)
└─ Multi-tenant SaaS ──→ API gateway + microservices + sharded DB
Cache Invalidation Decision Tree
Data changed? ─┬─ Rarely (config) ──→ TTL-based expiration (24h)
├─ Frequently (feed) ──→ Write-through + event-driven invalidation
├─ User-specific ──→ Per-user cache key + broadcast invalidation
└─ Global + must-be-fresh ──→ Write-through cache + CDN purge
Scaling Decision Tree
System slow? ─┬─ CPU bound ──→ Scale out (add instances behind LB)
├─ Memory bound ──→ Add cache layer (Redis/Memcached)
├─ I/O bound ──→ Add read replicas + connection pooling
├─ DB lock contention ──→ Shard data + CQRS
└─ Network bound ──→ Compression + CDN + keep-alive pooling
10. Testing Strategy
Load Testing with k6
// system-design-load-test.js
import http from 'k6/http';
import { check, sleep } from 'k6';
export const options = {
scenarios: {
// Steady-state: 80% of traffic
steady: {
executor: 'constant-arrival-rate',
rate: 2000,
timeUnit: '1s',
duration: '5m',
preAllocatedVUs: 100,
},
// Spike: 20% burst traffic
spike: {
executor: 'ramping-arrival-rate',
startRate: 0,
timeUnit: '1s',
preAllocatedVUs: 50,
stages: [
{ duration: '30s', target: 5000 }, // Ramp up
{ duration: '1m', target: 5000 }, // Sustain
{ duration: '30s', target: 0 }, // Ramp down
],
},
},
thresholds: {
http_req_duration: ['p(95)<200', 'p(99)<500'],
http_req_failed: ['rate<0.001'],
},
};
export default function () {
const res = http.get('http://api.myapp.com/v1/products', {
headers: { Authorization: Bearer ${__ENV.API_TOKEN} },
});
check(res, {
'status 200': (r) => r.status === 200,
'response < 200ms': (r) => r.timings.duration < 200,
});
sleep(0.5);
}
Contract Testing (PACT)
// Consumer contract test
const { Pact } = require('@pact-foundation/pact');
const provider = new Pact({
consumer: 'OrderService',
provider: 'PaymentService',
});
describe('Payment API', () => {
beforeAll(() => provider.setup());
afterAll(() => provider.finalize());
it('should process payment', async () => {
await provider.addInteraction({
state: 'payment method exists',
uponReceiving: 'a request to charge',
withRequest: { method: 'POST', path: '/charge', headers: { 'Content-Type': 'application/json' }, body: { amount: 99.99 } },
willRespondWith: { status: 200, headers: { 'Content-Type': 'application/json' }, body: { status: 'charged', transaction_id: like('txn_abc123') } },
});
const result = await chargePayment(99.99);
expect(result.status).toBe('charged');
});
});
Connection Pooling
# Async connection pool (asyncpg)
import asyncpg
async def get_pool():
return await asyncpg.create_pool(
dsn='postgresql://app:pass@db:5432/app_db',
min_size=5, # Keep warm connections
max_size=20, # Max concurrent connections
max_inactive_connection_lifetime=300, # 5min idle timeout
command_timeout=30, # 30s query timeout
)
Redis connection pool
import redis.asyncio as aioredis
redis_pool = aioredis.ConnectionPool(
host='redis', port=6379, db=0,
max_connections=50,
decode_responses=True,
socket_keepalive=True,
socket_connect_timeout=5,
)
B-Tree vs LSM-Tree Decision
Index type? ─┬─ Point lookups dominant ──→ B-Tree (PostgreSQL default)
├─ Range scans common ──→ B-Tree with covering index
├─ Write-heavy, append-only ──→ LSM-Tree (Cassandra, RocksDB)
└─ Time-series data ──=> LSM + compaction (InfluxDB, TimescaleDB)
Security Architecture
Defense in Depth Layers
Internet ──→ CDN/WAF ──→ Load Balancer ──→ Reverse Proxy ──→ App Server ──→ Database
│ │ │ │ │ │
│ DDoS filter TLS termination Rate limiting Auth/JWT Encryption
│ Bot detection HSTS headers Input validation RBAC At-rest
│ Geo-blocking CSP headers CSRF tokens Audit logs In-transit
Zero Trust Checklist
Never trust, always verify — authenticate every request, even internal
Least privilege — each service gets minimum permissions (IAM roles)
Microsegmentation — each service in its own network zone
Encryption everywhere — TLS in transit, AES-256 at rest
Secret management — Vault/AWS Secrets Manager, never in env vars
Audit logging — every API call logged with correlation ID
Automated rotation — keys/certs rotate every 30-90 days
HashiCorp Vault Integration
# Start Vault dev server (testing only)
vault server -dev
Production: file or consul storage
vault server -config=/etc/vault.d/vault.hcl
Enable secrets engine
vault secrets enable -path=secret kv-v2
Store database credentials
vault kv put secret/database \
username="app_user" \
password="$(openssl rand -base64 32)" \
connection_string="postgresql://app:**@db:5432/app_db"
Application reads credentials at runtime
vault kv get -field=password secret/database
Dynamic database credentials (auto-rotated)
vault secrets enable database
vault write database/config/app_db \
plugin_name=postgresql-database-plugin \
allowed_roles="app-role" \
connection_url="postgresql://{{username}}:{{password}}@db:5432/app_db" \
username="vault_admin" \
password="vault_admin_pass"
vault write database/roles/app-role \
db_name=app_db \
creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'" \
default_ttl="1h" \
max_ttl="24h"
Deployment Strategies
Blue-Green Deployment
┌──────────────┐ ┌──────────────┐
│ Blue (v1) │ │ Green (v2) │
│ Live Traffic│ │ Staging │
└──────┬───────┘ └──────┬───────┘
│ │
└──────┬──────────────┘
│
Load Balancer
(switch target)
Steps:
Deploy v2 to Green environment
Run smoke tests on Green
Switch LB from Blue → Green
Monitor error rate for 15min
Rollback: switch LB back to Blue if errors
Blue becomes next staging environment
Canary Deployment (Progressive)
# Nginx canary: route 5% traffic to v2
upstream backend {
server 10.0.1.1:8080 weight=95; # v1 (stable)
server 10.0.1.2:8080 weight=5; # v2 (canary)
}
Istio canary: VirtualService routing
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
spec:
http:
- match:
x-canary:
exact: "true"
route:
- destination:
host: app-service
subset: canary
- route:
- destination:
host: app-service
subset: stable
weight: 95
- destination:
host: app-service
subset: canary
weight: 5
Rolling Deployment with Health Checks
#!/bin/bash
Zero-downtime rolling deployment
NEW_VERSION=$1
HEALTH_URL="http://localhost:8080/health"
INSTANCES=(10.0.1.1 10.0.1.2 10.0.1.3)
for INSTANCE in "${INSTANCES[@]}"; do
echo "Deploying $NEW_VERSION to $INSTANCE..."
# Take instance out of LB
curl -X POST "http://lb:8080/api/backends/$INSTANCE/drain"
sleep 5
# Deploy
ssh $INSTANCE "docker pull app:$NEW_VERSION && docker stop app && docker run -d --name app app:$NEW_VERSION"
# Wait for health check
for i in $(seq 1 30); do
if curl -sf "$INSTANCE:8080/health" | grep -q "healthy"; then
echo "✅ $INSTANCE healthy"
break
fi
sleep 2
done
# Add back to LB
curl -X POST "http://lb:8080/api/backends/$INSTANCE/enable"
sleep 10 # Let it warm up
done
echo "✅ Rolling deployment complete"
HTTP/2 Server Push — send critical resources before browser asks
TCP Fast Open — reduce TCP handshake: sysctl net.ipv4.tcp_fastopen=3
TLS Session Resumption — session tickets reduce handshake from RTT×2 to 0
Keep-Alive Connections — reuse connections, reduce TCP overhead (Nginx: keepalive_timeout 65)
Compression — gzip/brotli for text assets, save 60-80% bandwidth
Connection Pooling — pre-warm DB/Redis connections, avoid cold starts
13. Best Practices
Start monolith, extract microservices only when you have clear bounded contexts
Design for 10x growth — but don't pre-optimize for 1000x
Every external call needs a circuit breaker, timeout, and retry
Cache invalidation is the hardest problem — use explicit invalidation over TTL where possible
Always provision read replicas before you need them
Use async everywhere — message queues for write-heavy paths
Monitor the RED metrics — Rate, Errors, Duration
Design idempotent APIs — retries are inevitable in distributed systems
Never share databases between services — each service owns its data
Document capacity limits explicitly: max QPS, max data size, max connections
13. Common Pitfalls
| # | Pitfall | Impact | Fix |
|---|---------|--------|-----|
| 1 | Premature microservices | Operational complexity explosion | Start monolith, split on domain boundaries |
| 2 | Cache without invalidation | Stale data, confused users | Write-through cache + event-driven invalidation |
| 3 | Shared database across services | Tight coupling, migration hell | Database-per-service + API contracts |
| 4 | No rate limiting | Thundering herd, cascading failures | LB rate limiting + API gateway quotas |
| 5 | Ignoring tail latencies | p99 >> p50, angry users | Hedge requests, timeout budgets |
| 6 | Synchronous call chains | Latency adds up, cascade failures | Async events, saga pattern |
| 7 | No connection pooling | Connection storms, DB overwhelm | Pool warm connections (min_size=5) |
| 8 | Hot partitions in sharding | Uneven load, one shard overwhelmed | Consistent hashing with vnodes |
| 9 | Missing correlation IDs | Can't trace across services | UUID per request, propagate in headers |
| 10 | Ignoring backpressure | Consumer overwhelmed, memory bloat | Buffer limits + dead letter queues |
14. Quick Reference
# Nginx: test config, reload, check status
nginx -t && nginx -s reload
ss -tlnp | grep nginx
HAProxy: check config, stats
haproxy -c -f /etc/haproxy/haproxy.cfg
curl http://localhost:8404/stats
Redis: memory, connections, slow log
redis-cli INFO memory | grep used_memory_human
redis-cli INFO clients | grep connected_clients
redis-cli SLOWLOG GET 10
Kafka: topic list, consumer lag, describe
kafka-topics --bootstrap-server kafka:9092 --list
kafka-consumer-groups --bootstrap-server kafka:9092 --describe --group orders
kafka-configs --bootstrap-server kafka:9092 --entity-type topics --describe
PostgreSQL: active connections, slow queries
SELECT * FROM pg_stat_activity WHERE state = 'active';
SELECT query, calls, total_time, mean_time FROM pg_stat_statements ORDER BY total_time DESC LIMIT 10;
MySQL: process list, slow queries, replication status
SHOW PROCESSLIST;
SELECT * FROM mysql.slow_log ORDER BY start_time DESC LIMIT 10;
SHOW SLAVE STATUS\G