Load Balancing Strategies & Failover Mechanisms

Your app is running on one web server, and it's handling traffic fine — until one day it isn't. Maybe there's a traffic spike, maybe the server needs a restart for a deploy, maybe it just crashes. Whatever the reason, with a single server, there's no such thing as a small failure. Everyone's request fails at once.

The fix everyone reaches for is "add a second server." That's the easy part. The hard part is: which server gets the next request, and what happens the moment one of them stops responding? That's the whole job of a load balancer, and the algorithm it uses to answer "which server" has real consequences depending on your traffic shape.

Load Balancing Algorithms

1. Round-Robin

Requests are distributed sequentially across servers.

• Pros: Simple, fair distribution, no state to track.
• Cons: Doesn't account for server capacity or current load. If one server is already handling a long-running request, it still gets the next one in line regardless.

2. Least Connections

Directs traffic to the server with the fewest active connections.

• Pros: Handles long-lived connections (WebSockets, streaming) much better than round-robin, since it actually reacts to current load.
• Cons: Requires tracking connection state, and it's not particularly meaningful for short-lived HTTP requests that open and close in milliseconds.

3. Weighted Round-Robin

Assigns weights to servers based on their capacity, so a bigger box gets proportionally more traffic than a smaller one.

• Pros: Accounts for heterogeneous hardware — useful during a rolling migration when half your fleet is on new, higher-spec instances.
• Cons: Someone has to actually maintain those weights as hardware changes, or they silently go stale.

4. Consistent Hashing

Hash the client IP or a request property to determine the backend server.

Hash(Client IP) % Number of Servers = Assigned Server

• Pros: The same client reliably lands on the same server — useful for session affinity without a shared session store.
• Cons: Plain modulo hashing falls apart the moment you add or remove a server, which is exactly the problem the "Virtual Nodes" section below solves.

Health Checks & Failover

Picking an algorithm assumes every server in the pool is healthy. The other half of a load balancer's job is figuring out when that stops being true.

Active Health Checks

The load balancer periodically sends requests (HTTP GET, TCP ping) to each backend server.

• Interval: Check every 1–10 seconds (tunable).
• Timeout: If no response within 2–5 seconds, mark the server as unhealthy.
• Grace Period: Remove a server from the pool only after consecutive failures (e.g., 3 in a row) — otherwise a single slow response yanks a perfectly healthy server out of rotation.

Passive Health Checks

The load balancer watches real request failures and marks servers unhealthy on the fly, instead of waiting for the next scheduled probe.

• Pros: Faster to react, since it's using live traffic instead of synthetic checks.
• Cons: Some real user requests fail before the server gets marked unhealthy — you're detecting the problem using actual customer impact.

Failover Strategy

• Cold Failover: Clients are rerouted to healthy servers immediately, no graceful handoff.
• Warm Failover: Existing connections are drained before a server is removed — better for anything mid-transaction.
• Hot Failover: Heartbeats or consensus detect failure before it ever reaches a client-facing request.

Consistent Hashing with Virtual Nodes

Here's the actual production problem consistent hashing fixes: with simple modulo hashing (hash(key) % N), adding or removing a single server changes N, which changes the result of every hash — meaning nearly all your keys suddenly map to a different server. For a cache layer, that's effectively a full cache wipe every time you scale.

Consistent hashing avoids this by:

1. Mapping servers onto a virtual Hash Ring (0 to $2^ - 1$).
2. For each request, hashing the client ID and finding the next server clockwise on the ring.
3. If that server fails, requests naturally fall through to the next healthy server on the ring — no full remap needed.

Virtual Nodes: Each physical server is represented by multiple points on the ring, not just one. Without virtual nodes, a ring with only a handful of physical servers distributes load unevenly by chance — some arcs of the ring end up much bigger than others. Virtual nodes smooth that out and also minimize how much data moves when a server joins or leaves.

Layer 4 vs. Layer 7 Load Balancing

Layer 4 (Transport Layer)

Load balancers make decisions based on IP address and TCP/UDP ports, without looking at the actual HTTP request.

• Speed: Very fast — there's no application data to parse.
• Limitation: Can't route based on request path, headers, or method.
• Use Case: Raw TCP/UDP streams, DNS.

Layer 7 (Application Layer)

Load balancers parse HTTP headers, URLs, and request bodies before deciding where to send a request.

• Flexibility: Route /api/* to your API servers and /images/* to a CDN, all from the same load balancer.
• Overhead: Slower, since it has to fully parse each request first.
• Use Case: Web applications, microservices — most modern web traffic.

Session Persistence (Sticky Sessions)

Here's a scenario I've seen play out more than once: an application stores session state in-process (in server memory, not a shared store), so it needs sticky sessions to make sure a client always lands on the same backend that has their data.

• Cookie-based: The load balancer injects a cookie identifying the backend, and future requests from that client route to the same server.
• IP-based: All requests from the same IP route to the same backend.
• Cons: This quietly breaks the whole point of having multiple servers. If that one server goes down, every user pinned to it loses their session. If it's popular, that server becomes a hot spot while others sit idle. And it makes autoscaling nearly useless, because you can't safely add or remove servers without disrupting sessions.

Better Approach: Move sessions out of the server entirely and into a shared store like Redis. Then any server can handle any request, sticky sessions become unnecessary, and you get the full benefit of horizontal scaling back.

Key Takeaways

• The load balancing algorithm you pick should match your traffic shape — round-robin for uniform short requests, least-connections for long-lived ones, weighted round-robin for mixed hardware.
• Active and passive health checks answer different questions: "is it healthy right now" (active) vs. "did it just fail on real traffic" (passive) — most production setups use both.
• Consistent hashing exists specifically to avoid a full remap every time you scale the server pool — plain modulo hashing breaks this in a way that's easy to overlook until you're operating at scale.
• Sticky sessions are almost always a symptom of storing session state in the wrong place. Moving sessions into Redis (or similar) removes the need for stickiness entirely and unlocks real horizontal scaling.

Next Steps

Once traffic is reliably routed to healthy servers, the next question is what those servers keep doing with every one of those requests — hitting the database for the same answers over and over. That's what caching fixes, covered next.