System Design Roadmap Part 2: Core Building Blocks – Load Balancers, CDN, Caching, API Gateway, DBs & Kafka

Introduction: The Scalability Puzzle

In Part 1: Networking Basics, we explored how bits and bytes travel across the internet. But what happens when your single backend server goes from handling 50 requests per second (RPS) to 500,000 RPS?

A single high-spec server (Vertical Scaling / Scaling Up) eventually hits hard physical constraints: CPU socket limits, RAM bus bandwidth, and catastrophic Single Points of Failure (SPOF). To survive modern web scale, systems must scale horizontally by stitching together independent, loosely coupled building blocks.

In this guide, we break down the 6 fundamental building blocks required to assemble any enterprise-scale distributed architecture.

1. Load Balancers: Distributing Traffic and Eliminating SPOFs

A Load Balancer (LB) sits between clients and server pools, distributing incoming network requests so that no single server becomes overloaded.

Layer 4 (Transport) vs. Layer 7 (Application) Load Balancing

Understanding the distinction between L4 and L7 load balancers is a core interview topic:

Routing Algorithms in Practice

1. Round Robin: Distributes requests sequentially across servers. Best when all backend servers have identical compute specs and requests require uniform processing time.
2. Weighted Round Robin: Assigns higher proportions of traffic to servers with superior CPU/RAM hardware.
3. Least Connections: Routes traffic to the server currently maintaining the fewest active open connections. Ideal for long-lived transactions or WebSocket connections.
4. IP Hash / Consistent Hash: Hashes the client's IP address to map them to the same backend server. Useful for stateful session persistence (though stateless designs with Redis sessions are preferred).

High Availability of the Load Balancer Itself

A common interview trap: "If all traffic goes through the load balancer, isn't the load balancer a Single Point of Failure?"

The Solution: Deploy Load Balancers in an Active-Passive or Active-Active pair using protocols like VRRP (Virtual Router Redundancy Protocol) or Keepalived. Both balancers share a floating Virtual IP (VIP). Heartbeat pings continuously check the Active node; if it stops responding, the Passive node promotes itself and assumes the VIP in sub-second time.

2. CDN (Content Delivery Network): Edge Caching & Low Latency

A CDN is a globally distributed network of Point of Presence (PoP) edge servers that cache static and semi-static assets (images, videos, JS, CSS bundles, HTML, API responses) close to end users.

Without CDN:  [User in Jakarta] ─── (240ms RTT) ───► [Origin Server in Virginia]
With CDN:     [User in Jakarta] ─── (12ms RTT)  ───► [Jakarta CDN Edge PoP]

Push vs. Pull CDN Architecture

Advanced CDN Strategies:

• Origin Shielding: An intermediate caching tier between global edge PoPs and your origin data center. If 50 Asian edge servers miss a cache item, they query the Singapore Origin Shield rather than overwhelming your primary origin with 50 redundant fetch calls.
• Cache Invalidation: Controlled via HTTP headers (Cache-Control: public, max-age=31536000, immutable). For rapid updates, use content-hashed asset URLs (bundle.a98f12.js) rather than expensive global CDN purges.

3. Caching (Redis): Accelerating Reads with In-Memory Storage

RAM is roughly 1,000 to 10,000 times faster than disk storage (NVMe SSD). In-memory caching layers like Redis or Memcached intercept read requests, slashing database load and dropping response latencies to single-digit milliseconds.

Caching Pitfalls in Production & How to Solve Them

1. Cache Stampede / Thundering Herd:
   • Problem: A heavily queried cache key (e.g., flash-sale product) expires. 10,000 concurrent threads detect the cache miss simultaneously and hammer the database with 10,000 identical SQL queries.
   • Solution: Use Distributed Mutex Locks (SET key value NX PX 5000) so only 1 worker rebuilds the cache, or implement Probabilistic Early Expiration (XFetch).
2. Cache Avalanche:
   • Problem: Thousands of cache keys are set with the exact same TTL (e.g., 3600s). At minute 60, all keys vanish simultaneously, dropping the entire traffic load onto the database.
   • Solution: Add randomized Jitter to TTLs (TTL = 3600 + rand(-300, 300)).
3. Cache Penetration:
   • Problem: An attacker queries non-existent IDs (/users/-99999). The key is never in cache, forcing a database lookup on every malicious request.
   • Solution: Cache null values with a short TTL (e.g., 60s), or place a Bloom Filter in front of the cache to instantly reject non-existent keys.

4. API Gateway: The Intelligent Traffic Controller

An API Gateway acts as the single reverse-proxy entry point for all client requests entering a microservices ecosystem.

Core Responsibilities:

1. Security & Auth Offloading: Validates JWT cryptographic signatures, checks user roles, and rejects expired tokens before requests reach downstream services. Internal services receive pre-validated identity headers (X-User-Id: 42).
2. Rate Limiting: Protects backend clusters against malicious scrapers or noisy neighbors using algorithms like Token Bucket or Redis-backed sliding windows.
3. SSL/TLS Termination: Handles expensive asymmetric cryptographic handshakes at the edge, allowing internal VPC communication over high-speed unencrypted HTTP or gRPC.
4. Backend-For-Frontend (BFF): Custom gateway facades tailored for different clients (e.g., mobile gateway aggregating 4 REST calls into 1 payload to conserve mobile battery).

5. Databases (SQL vs. NoSQL): Choosing the Right Storage Model

The database is usually the hardest component to scale horizontally because it maintains persistent state. Selecting the wrong database model early in an architecture is an expensive mistake.

The NoSQL Taxonomy Matrix

Indexing: B-Tree vs. LSM-Tree

• B-Trees / B+ Trees (PostgreSQL, MySQL InnoDB): Kept in balanced hierarchical pages. Highly optimized for fast point lookups and range queries, but writes incur random disk I/O to maintain balance.
• LSM-Trees (Log-Structured Merge-Trees) (Cassandra, RocksDB): Writes append sequentially to an in-memory MemTable and commit log, then flush to immutable disk SSTables. Delivers extraordinary write throughput, trading off slightly slower reads.

6. Message Queues & Event Streaming: Decoupled Systems with Kafka

Synchronous HTTP calls (Service A -> Service B -> Service C) suffer from Cascading Failures. If Service C experiences 5-second latency or crashes, Service A runs out of HTTP thread pool workers and collapses.

Asynchronous messaging decouples services in both time and space.

SYNCHRONOUS (Tight Coupling, Fragile):
[ Checkout ] ──HTTP POST──► [ Payment ] ──HTTP POST──► [ Notification ] ──► [ Inventory ]
(If Notification fails or hangs, entire checkout fails!)

ASYNCHRONOUS (Loose Coupling, Resilient):
[ Checkout ] ──► [ Payment ] ──► Emits "OrderCreated" event to [ Kafka Topic ]
                                                                      │
                                     ┌────────────────────────────────┼───────────────────────┐
                                     ▼                                ▼                       ▼
                             [ Notification ]                 [ Inventory ]           [ Analytics ]

Traditional Message Queue (RabbitMQ) vs. Distributed Event Log (Kafka)

Kafka Core Concepts in System Design:

1. Topic: A category or feed name to which records are published.
2. Partition: Topics are sliced into partitions distributed across cluster brokers. Each partition is an append-only ordered commit log.
3. Ordering Guarantee: Kafka guarantees strict message ordering within a single partition, but NOT across different partitions. To preserve order for a specific entity (e.g., a customer's order events), use the entity's ID as the Kafka Message Key.
4. Consumer Groups: Multiple consumer instances read from the same topic in parallel. Each partition is assigned to exactly one consumer in the group, enabling horizontal processing scaling.

Architecture Synthesis: Putting It All Together

Here is how these 6 building blocks cooperate during a single high-scale event:

1. Client requests "Checkout" via Mobile App.
2. Anycast DNS routes request to the nearest Regional Load Balancer (L4 NLB).
3. NLB passes TCP connection to L7 Application Load Balancer (ALB / Envoy).
4. ALB terminates SSL and forwards to API Gateway.
5. API Gateway authenticates user JWT, validates rate-limiting quota in Redis,
   and routes request to the Order Service.
6. Order Service updates PostgreSQL with strict ACID transactions.
7. Order Service invalidates the user's cached cart in Redis.
8. Order Service publishes an "OrderPlaced" event to Apache Kafka.
9. Downstream Notification Service, Fraud Detection Service, and Analytics Service
   consume the Kafka event independently at their own pace without impacting
   the client response latency!

What's Next in Part 3?

Now that you command the 6 core building blocks of distributed architecture, how do you handle scale when a database becomes too big for a single machine, or when network partitions threaten data consistency?

In Part 3: Distributed Patterns & Concepts, we delve into:

• CAP Theorem & PACELC Theorem
• Consistent Hashing (Hash rings, virtual nodes, and zero-downtime cluster rebalancing)
• Database Sharding (Range, directory, and hash sharding strategies)
• Idempotency (Preventing double-charges across retried networks)
• Distributed Microservice Patterns (Transactional Outbox, Saga Pattern, and CQRS)
• Rate Limiting Algorithms (Token Bucket, Leaky Bucket, Sliding Window)