System Design Mastery

System Design Mastery

1. System Design Framework

Four-Step Framework

Step 1: REQUIREMENTS ──→ Functional + Non-Functional (QPS, latency, storage, availability)
Step 2: ESTIMATION ───→ Back-of-envelope: users × requests × data size × redundancy
Step 3: HIGH-LEVEL ───→ Draw architecture with 5-7 boxes (client → LB → API → DB → cache)
Step 4: DEEP-DIVE ────→ Pick 2-3 components, discuss tradeoffs, bottlenecks, alternatives

Estimation Cheat Sheet

2. CAP Theorem & Consistency Models

CAP Theorem: Choose 2 of 3
┌─────────────────────────────────────────────────────────────┐
│                   CAP Tradeoffs                              │
│                                                              │
│  AP (Availability + Partition Tolerance)                    │
│  └─ DynamoDB, Cassandra, CouchDB — eventual consistency      │
│                                                              │
│  CP (Consistency + Partition Tolerance)                     │
│  └─ ZooKeeper, etcd, HBase — unavailable during partition   │
│                                                              │
│  CA (Consistency + Availability) — impossible in distributed│
│  └─ RDBMS (single-node, no partition tolerance)              │
└─────────────────────────────────────────────────────────────┘

PACELC Extension

3. Load Balancing

Decision Tree

Need load balancing? ─┬─ Layer 4 (TCP/UDP) ──→ HAProxy, NLB, iptables
                       │   Fast, no HTTP awareness
                       ├─ Layer 7 (HTTP) ──→ Nginx, ALB, Traefik, Kong
                       │   Path routing, SSL termination, rate limiting
                       └─ Global (DNS) ──→ Cloudflare, Route 53, GeoDNS
                           Latency-based routing, DDoS protection

Nginx Load Balancer Config

upstream backend {
    least_conn;                    # Algorithm: round_robin | least_conn | ip_hash | random
    server 10.0.1.1:8080 weight=5;
    server 10.0.1.2:8080 weight=3;
    server 10.0.1.3:8080 backup;   # Only if primaries fail
    
    keepalive 32;                  # Connection pooling
}
server {
    listen 443 ssl http2;
    ssl_certificate /etc/ssl/cert.pem;
    ssl_certificate_key /etc/ssl/key.pem;
    
    location / {
        proxy_pass http://backend;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Host $host;
        
        # Health check (active)
        proxy_next_upstream error timeout http_502 http_503;
        proxy_connect_timeout 5s;
        proxy_read_timeout 30s;
    }
    
    # Health check endpoint (passive)
    location /health {
        proxy_pass http://backend/health;
        access_log off;
    }
}

HAProxy TCP Load Balancer

# /etc/haproxy/haproxy.cfg
global
    maxconn 100000
    daemon
defaults
    mode tcp
    timeout connect 5s
    timeout client 30s
    timeout server 30s
    retries 3
frontend mysql_front
    bind *:3306
    default_backend mysql_back
backend mysql_back
    balance leastconn
    option mysql-check user haproxy_check
    server mysql1 10.0.1.1:3306 check inter 5s rise 3 fall 2
    server mysql2 10.0.1.2:3306 check inter 5s rise 3 fall 2 backup

4. Caching

Cache Strategy Decision Tree

Need caching? ─┬─ Read-heavy, static data ──→ Cache-Aside (lazy load)
                ├─ Write-heavy, must be fresh ──→ Write-Through (sync write)
                ├─ Batch writes acceptable ──→ Write-Behind (async flush)
                └─ Session/state data ──→ Distributed cache (Redis Cluster)

Redis Cache-Aside Pattern

import redis
import json
r = redis.Redis(host='redis-primary', port=6379, db=0, decode_responses=True)
async def get_user(user_id: str) -> dict:
    """Cache-aside: check cache first, miss → fetch from DB → populate cache"""
    cache_key = f"user:{user_id}"
    
    # 1. Check cache
    cached = r.get(cache_key)
    if cached:
        return json.loads(cached)
    
    # 2. Cache miss → fetch from DB
    user = await db.fetch_one("SELECT * FROM users WHERE id = $1", user_id)
    
    # 3. Populate cache with TTL
    r.setex(cache_key, 3600, json.dumps(user))  # 1-hour TTL
    
    return user
async def invalidate_user(user_id: str):
    """Delete cached entry on update"""
    r.delete(f"user:{user_id}")
    r.delete(f"user_posts:{user_id}")  # Invalidate related caches

Redis Cluster Config

# redis.conf for each node (6-node cluster: 3 masters + 3 replicas)
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 15000
cluster-announce-ip 10.0.1.1
cluster-announce-port 6379
cluster-announce-bus-port 16379
Create cluster
redis-cli --cluster create \
  10.0.1.1:6379 10.0.1.2:6379 10.0.1.3:6379 \
  10.0.1.4:6379 10.0.1.5:6379 10.0.1.6:6379 \
  --cluster-replicas 1

5. Database Sharding & Replication

Sharding Decision Tree

Data too large for one node? ─┬─ No ──→ Vertical scaling (bigger instance)
                                ├─ Yes, <1TB ──→ Read replicas first
                                ├─ Yes, 1-10TB ──→ Range or Hash sharding
                                └─ Yes, >10TB ──→ Vitess/CockroachDB (auto-sharding)

Hash-Based Sharding

def get_shard(key: str, num_shards: int = 16) -> int:
    """Consistent hash for shard routing"""
    return int(hashlib.md5(key.encode()).hexdigest(), 16) % num_shards
Resharding with virtual nodes (256 vnodes per physical node)
RING = ConsistentHash(vnodes=256, replicas=3)
RING.add_node('shard-0', 'db-shard-0.internal:5432')
RING.add_node('shard-1', 'db-shard-1.internal:5432')
RING.add_node('shard-2', 'db-shard-2.internal:5432')
Only ~25% of keys move when adding a 4th node (vs 75% with naive modulo)

MySQL Master-Slave Replication

# my.cnf (master)
[mysqld]
server-id = 1
log-bin = mysql-bin
binlog-format = ROW
gtid-mode = ON
enforce-gtid-consistency = ON
binlog-do-db = app_db
sync-binlog = 1
innodb-flush-log-at-trx-commit = 1
my.cnf (replica)
[mysqld]
server-id = 2
relay-log = relay-bin
read-only = 1
gtid-mode = ON
enforce-gtid-consistency = ON

6. Message Queues & Event Streaming

Queue Selection Decision Tree

Which messaging system? ─┬─ Simple task queue, <10K msg/s ──→ Redis Streams / SQS
                          ├─ Pub/Sub, fan-out, routing ──→ RabbitMQ
                          ├─ High throughput, event streaming ──→ Apache Kafka
                          └─ Exactly-once, transactions ──→ Kafka + Transactions API

Kafka Producer (Python)

from confluent_kafka import Producer, KafkaError
conf = {
    'bootstrap.servers': 'kafka-1:9092,kafka-2:9092,kafka-3:9092',
    'client.id': 'order-service',
    'acks': 'all',                    # Wait for all ISR acks
    'retries': 3,
    'retry.backoff.ms': 1000,
    'enable.idempotence': True,       # Exactly-once semantics
    'compression.type': 'lz4',
    'linger.ms': 50,                  # Batch small messages
    'batch.size': 32768,
}
producer = Producer(conf)
def delivery_report(err, msg):
    if err:
        print(f"Delivery failed: {err}")
    else:
        print(f"Delivered to {msg.topic()} [{msg.partition()}] @ offset {msg.offset()}")
Produce message
producer.produce(
    topic='orders',
    key=str(order_id).encode(),
    value=json.dumps(order).encode(),
    headers={'source': b'order-service', 'trace_id': trace_id.encode()},
    callback=delivery_report
)
producer.flush()

RabbitMQ with Dead Letter Queue

# Consumer config with DLQ
spring:
  rabbitmq:
    host: rabbitmq
    listener:
      simple:
        acknowledge-mode: manual
        prefetch: 10
        retry:
          max-attempts: 3
          max-interval: 5s
Queue declaration with DLQ
@Queue(value = "orders", arguments = {
  @Argument(name = "x-dead-letter-exchange", value = "dlq.exchange"),
  @Argument(name = "x-dead-letter-routing-key", value = "orders.dlq"),
  @Argument(name = "x-message-ttl", value = "86400000")  # 24h TTL
})

7. API Gateway Patterns

Kong Rate Limiting Config

# Kong declarative config (kong.yml)
_format_version: "3.0"
services:
  - name: order-service
    url: http://order-service:8080
    routes:
      - name: orders
        paths:
          - /api/orders
        methods:
          - GET
          - POST
    plugins:
      - name: rate-limiting
        config:
          minute: 100
          hour: 5000
          policy: redis
          redis_host: redis
          redis_port: 6379
          redis_database: 1
      - name: key-auth
        config:
          key_names:
            - apikey
      - name: request-transformer
        config:
          add:
            headers:
              - "X-Gateway-Timestamp:$(date)"

8. Microservices Patterns

Circuit Breaker (Python)

from circuitbreaker import circuit
@circuit(failure_threshold=5, recovery_timeout=30, expected_exception=ConnectionError)
async def call_payment_service(order_id: str):
    """Circuit breaker wraps external service calls"""
    async with httpx.AsyncClient() as client:
        response = await client.post(
            f"http://payment-service:8080/charge",
            json={"order_id": order_id},
            timeout=5.0
        )
        response.raise_for_status()
        return response.json()
Fallback when circuit is open
@circuit(fallback_function=lambda: {"status": "pending", "reason": "service_unavailable"})
async def call_payment_with_fallback(order_id: str):
    return await call_payment_service(order_id)

Saga Pattern (Choreography)

Order Flow (Saga Choreography):
┌──────────────┐    ┌──────────────┐    ┌──────────────┐    ┌──────────────┐
│   Order Svc   │───→│  Payment Svc  │───→│  Inventory   │───→│  Shipping     │
│  (created)    │    │  (charged)    │    │  (reserved)  │    │  (scheduled)  │
└──────┬───────┘    └──────┬───────┘    └──────┬───────┘    └──────────────┘
       │                   │                    │
       │  COMPENSATE      │  COMPENSATE        │  COMPENSATE
       ↓                  ↓                    ↓
  Cancel Order      Refund Payment      Release Inventory
Each step publishes an event; next service subscribes and reacts.
If any step fails, compensating transactions fire in reverse order.

9. Flowcharts & Decision Diagrams

Architecture Selection Decision Tree

What are you building? ──┬─ CRUD web app <1K QPS ──→ Monolith (Rails/Django/NestJS)
                         ├─ Real-time <10K connections ──→ WebSocket server (Socket.IO/ws)
                         ├─ High throughput streaming ──→ Kafka + microservices
                         ├─ Compute-heavy jobs ──→ Task queue (Celery/BullMQ)
                         └─ Multi-tenant SaaS ──→ API gateway + microservices + sharded DB

Cache Invalidation Decision Tree

Data changed? ─┬─ Rarely (config) ──→ TTL-based expiration (24h)
                ├─ Frequently (feed) ──→ Write-through + event-driven invalidation
                ├─ User-specific ──→ Per-user cache key + broadcast invalidation
                └─ Global + must-be-fresh ──→ Write-through cache + CDN purge

Scaling Decision Tree

System slow? ─┬─ CPU bound ──→ Scale out (add instances behind LB)
               ├─ Memory bound ──→ Add cache layer (Redis/Memcached)
               ├─ I/O bound ──→ Add read replicas + connection pooling
               ├─ DB lock contention ──→ Shard data + CQRS
               └─ Network bound ──→ Compression + CDN + keep-alive pooling

10. Testing Strategy

Load Testing with k6

// system-design-load-test.js
import http from 'k6/http';
import { check, sleep } from 'k6';
export const options = {
  scenarios: {
    // Steady-state: 80% of traffic
    steady: {
      executor: 'constant-arrival-rate',
      rate: 2000,
      timeUnit: '1s',
      duration: '5m',
      preAllocatedVUs: 100,
    },
    // Spike: 20% burst traffic
    spike: {
      executor: 'ramping-arrival-rate',
      startRate: 0,
      timeUnit: '1s',
      preAllocatedVUs: 50,
      stages: [
        { duration: '30s', target: 5000 },  // Ramp up
        { duration: '1m', target: 5000 },   // Sustain
        { duration: '30s', target: 0 },      // Ramp down
      ],
    },
  },
  thresholds: {
    http_req_duration: ['p(95)<200', 'p(99)<500'],
    http_req_failed: ['rate<0.001'],
  },
};
export default function () {
  const res = http.get('http://api.myapp.com/v1/products', {
    headers: { Authorization: Bearer ${__ENV.API_TOKEN} },
  });
  check(res, {
    'status 200': (r) => r.status === 200,
    'response < 200ms': (r) => r.timings.duration < 200,
  });
  sleep(0.5);
}

Contract Testing (PACT)

// Consumer contract test
const { Pact } = require('@pact-foundation/pact');
const provider = new Pact({
  consumer: 'OrderService',
  provider: 'PaymentService',
});
describe('Payment API', () => {
  beforeAll(() => provider.setup());
  afterAll(() => provider.finalize());
it('should process payment', async () => {
    await provider.addInteraction({
      state: 'payment method exists',
      uponReceiving: 'a request to charge',
      withRequest: { method: 'POST', path: '/charge', headers: { 'Content-Type': 'application/json' }, body: { amount: 99.99 } },
      willRespondWith: { status: 200, headers: { 'Content-Type': 'application/json' }, body: { status: 'charged', transaction_id: like('txn_abc123') } },
    });
    const result = await chargePayment(99.99);
    expect(result.status).toBe('charged');
  });
});

11. Performance Optimization

Connection Pooling

# Async connection pool (asyncpg)
import asyncpg
async def get_pool():
    return await asyncpg.create_pool(
        dsn='postgresql://app:pass@db:5432/app_db',
        min_size=5,          # Keep warm connections
        max_size=20,         # Max concurrent connections
        max_inactive_connection_lifetime=300,  # 5min idle timeout
        command_timeout=30,   # 30s query timeout
    )
Redis connection pool
import redis.asyncio as aioredis
redis_pool = aioredis.ConnectionPool(
    host='redis', port=6379, db=0,
    max_connections=50,
    decode_responses=True,
    socket_keepalive=True,
    socket_connect_timeout=5,
)

B-Tree vs LSM-Tree Decision

Index type? ─┬─ Point lookups dominant ──→ B-Tree (PostgreSQL default)
              ├─ Range scans common ──→ B-Tree with covering index
              ├─ Write-heavy, append-only ──→ LSM-Tree (Cassandra, RocksDB)
              └─ Time-series data ──=> LSM + compaction (InfluxDB, TimescaleDB)

Security Architecture

Defense in Depth Layers

Internet ──→ CDN/WAF ──→ Load Balancer ──→ Reverse Proxy ──→ App Server ──→ Database
  │            │              │                  │                 │              │
  │         DDoS filter   TLS termination   Rate limiting     Auth/JWT      Encryption
  │         Bot detection  HSTS headers     Input validation  RBAC          At-rest
  │         Geo-blocking   CSP headers      CSRF tokens       Audit logs    In-transit

Zero Trust Checklist

HashiCorp Vault Integration

# Start Vault dev server (testing only)
vault server -dev
Production: file or consul storage
vault server -config=/etc/vault.d/vault.hcl
Enable secrets engine
vault secrets enable -path=secret kv-v2
Store database credentials
vault kv put secret/database \
  username="app_user" \
  password="$(openssl rand -base64 32)" \
  connection_string="postgresql://app:**@db:5432/app_db"
Application reads credentials at runtime
vault kv get -field=password secret/database
Dynamic database credentials (auto-rotated)
vault secrets enable database
vault write database/config/app_db \
  plugin_name=postgresql-database-plugin \
  allowed_roles="app-role" \
  connection_url="postgresql://{{username}}:{{password}}@db:5432/app_db" \
  username="vault_admin" \
  password="vault_admin_pass"
vault write database/roles/app-role \
  db_name=app_db \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'" \
  default_ttl="1h" \
  max_ttl="24h"

Deployment Strategies

Blue-Green Deployment

┌──────────────┐     ┌──────────────┐
│  Blue (v1)   │     │  Green (v2)  │
│  Live Traffic│     │   Staging    │
└──────┬───────┘     └──────┬───────┘
       │                     │
       └──────┬──────────────┘
              │
         Load Balancer
         (switch target)
Steps:
Deploy v2 to Green environment

Run smoke tests on Green

Switch LB from Blue → Green

Monitor error rate for 15min

Rollback: switch LB back to Blue if errors

Blue becomes next staging environment

Canary Deployment (Progressive)

# Nginx canary: route 5% traffic to v2
upstream backend {
    server 10.0.1.1:8080 weight=95;   # v1 (stable)
    server 10.0.1.2:8080 weight=5;     # v2 (canary)
}
Istio canary: VirtualService routing
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
spec:
  http:
  - match:
    - headers:
        x-canary:
          exact: "true"
    route:
    - destination:
        host: app-service
        subset: canary
  - route:
    - destination:
        host: app-service
        subset: stable
      weight: 95
    - destination:
        host: app-service
        subset: canary
      weight: 5

Rolling Deployment with Health Checks

#!/bin/bash
Zero-downtime rolling deployment
NEW_VERSION=$1
HEALTH_URL="http://localhost:8080/health"
INSTANCES=(10.0.1.1 10.0.1.2 10.0.1.3)
for INSTANCE in "${INSTANCES[@]}"; do
    echo "Deploying $NEW_VERSION to $INSTANCE..."
    
    # Take instance out of LB
    curl -X POST "http://lb:8080/api/backends/$INSTANCE/drain"
    sleep 5
    
    # Deploy
    ssh $INSTANCE "docker pull app:$NEW_VERSION && docker stop app && docker run -d --name app app:$NEW_VERSION"
    
    # Wait for health check
    for i in $(seq 1 30); do
        if curl -sf "$INSTANCE:8080/health" | grep -q "healthy"; then
            echo "✅ $INSTANCE healthy"
            break
        fi
        sleep 2
    done
    
    # Add back to LB
    curl -X POST "http://lb:8080/api/backends/$INSTANCE/enable"
    sleep 10  # Let it warm up
done
echo "✅ Rolling deployment complete"

Performance Optimization — Connection & Protocol

13. Best Practices

13. Common Pitfalls

| # | Pitfall | Impact | Fix |
|---|---------|--------|-----|
| 1 | Premature microservices | Operational complexity explosion | Start monolith, split on domain boundaries |
| 2 | Cache without invalidation | Stale data, confused users | Write-through cache + event-driven invalidation |
| 3 | Shared database across services | Tight coupling, migration hell | Database-per-service + API contracts |
| 4 | No rate limiting | Thundering herd, cascading failures | LB rate limiting + API gateway quotas |
| 5 | Ignoring tail latencies | p99 >> p50, angry users | Hedge requests, timeout budgets |
| 6 | Synchronous call chains | Latency adds up, cascade failures | Async events, saga pattern |
| 7 | No connection pooling | Connection storms, DB overwhelm | Pool warm connections (min_size=5) |
| 8 | Hot partitions in sharding | Uneven load, one shard overwhelmed | Consistent hashing with vnodes |
| 9 | Missing correlation IDs | Can't trace across services | UUID per request, propagate in headers |
| 10 | Ignoring backpressure | Consumer overwhelmed, memory bloat | Buffer limits + dead letter queues |

14. Quick Reference

# Nginx: test config, reload, check status
nginx -t && nginx -s reload
ss -tlnp | grep nginx
HAProxy: check config, stats
haproxy -c -f /etc/haproxy/haproxy.cfg
curl http://localhost:8404/stats
Redis: memory, connections, slow log
redis-cli INFO memory | grep used_memory_human
redis-cli INFO clients | grep connected_clients
redis-cli SLOWLOG GET 10
Kafka: topic list, consumer lag, describe
kafka-topics --bootstrap-server kafka:9092 --list
kafka-consumer-groups --bootstrap-server kafka:9092 --describe --group orders
kafka-configs --bootstrap-server kafka:9092 --entity-type topics --describe
PostgreSQL: active connections, slow queries
SELECT * FROM pg_stat_activity WHERE state = 'active';
SELECT query, calls, total_time, mean_time FROM pg_stat_statements ORDER BY total_time DESC LIMIT 10;
MySQL: process list, slow queries, replication status
SHOW PROCESSLIST;
SELECT * FROM mysql.slow_log ORDER BY start_time DESC LIMIT 10;
SHOW SLAVE STATUS\G