One server is a ceiling and a single point of failure. The fix — run several and put something in front to spread the traffic — sounds like the easiest decision in system design, and mostly it is. The interesting part is not the load balancer. It is what running several servers demands of your application, which is where designs actually fail.
This post covers both: how a load balancer picks a server, and what “stateless” really requires. Plus the layer above, where the fastest request is the one that never reaches you at all.
What it does, and the job people forget
┌──> [ app 1 ] ✓ healthy
clients ──> [ LB ] ────┼──> [ app 2 ] ✓ healthy
└──> [ app 3 ] ✗ failing health checks
— taken out of rotationThe obvious job is distribution. The job that matters more is the annotation on the third line: a load balancer is also the thing that notices a server is sick and stops sending it traffic.
Without health checks, losing one instance out of three does not cost you a third of your capacity — it costs you a third of your requests, as errors, until a human intervenes. With them, the same failure is invisible to users. That is the availability argument, and it is a stronger reason to run a load balancer than throughput is.
It also gives you two things you did not ask for and will use constantly: a place to terminate TLS, so certificates live in one place rather than on every server; and the ability to deploy without downtime, by draining one instance at a time.
Layer 4 or layer 7
The first real choice, and it is about how much the balancer understands.
A layer 4 balancer works at the TCP level. It sees IP addresses and ports, picks a backend, and forwards packets. It never looks inside the connection, which means it is very fast and cannot do very much.
A layer 7 balancer terminates the HTTP request, reads it, and then decides. It knows the path, the headers, the cookies and the method.
| Layer 4 | Layer 7 | |
|---|---|---|
| Sees | IP, port, TCP | URL, headers, cookies, method |
| Routing by path | No | Yes — /api/* here, /static/* there |
| TLS termination | No (passes through) | Yes |
| Retry a failed request | No — it does not know what a request is | Yes, safely, for idempotent methods |
| Throughput | Higher | Lower — it parses everything |
| Health checks | Is the port open? | Does GET /health return 200? |
For an HTTP API, layer 7 is nearly always the answer, and the health check row is why. “Is the port open?” is a much weaker question than “does this instance actually work?” A process that has deadlocked, lost its database connection, or run out of heap still accepts TCP connections perfectly happily.
Where it sits, physically
One more distinction that comes up: a load balancer can be hardware, software, or somebody else’s problem.
| Examples | When it fits | |
|---|---|---|
| Managed | AWS ALB/NLB, GCP LB, Cloudflare | Almost always. Already redundant across zones, scales itself, someone else is on call for it. |
| Software | nginx, HAProxy, Envoy, Traefik | Self-hosted, or when you need routing rules a managed one cannot express |
| Hardware | F5, Citrix | Existing datacenters, regulated environments. Rare in new designs. |
“A managed load balancer” is a perfectly good interview answer, provided you can say what it is doing for you — which is the redundancy from the section below, plus TLS termination, plus health checking. Naming the product is not the answer; knowing which of its jobs you depend on is.
Algorithms
How the balancer picks. There are five worth knowing and the differences only show up under load.
ROUND ROBIN 1, 2, 3, 1, 2, 3 ...
simple; assumes every request costs the same
WEIGHTED server 1 gets 2x the share of server 2
for a mixed fleet
LEAST CONNECTIONS send to whoever has fewest in flight
self-correcting when request costs vary
LEAST RESPONSE TIME fewest connections, tie-broken by latency
routes around a sick-but-alive instance
IP HASH / CONSISTENT hash(client) -> always the same server
needed only when something is NOT statelessRound robin is the sensible default and its assumption is the thing to remember: it assumes requests cost roughly the same. If one endpoint takes 5ms and another takes two seconds, round robin will happily give one server three of the slow ones in a row.
Least connections fixes that without needing to know anything about the endpoints, which is why it is the right default for a mixed API.
IP hash deserves suspicion. It exists to send the same client to the same server every time, and the only reason you need that is that your servers are keeping something in memory — which is the problem, not the solution. It is also unfair, because thousands of users behind one corporate NAT all hash to the same instance.
Health checks that actually check health
The gap between “the process is running” and “this instance can serve requests” is where availability is lost.
A check that returns a constant is worse than useless, because it will keep reporting healthy while the database connection pool is exhausted. A check should touch the dependencies it needs — and, critically, distinguish the ones it needs from the ones it merely likes.
StayHub’s reports each dependency separately and lets only one of them decide:
es_ok = es_available()
cache_ok = cache.available()
return Health(
status="ok" if db_ok else "degraded",
database=db_ok,
elasticsearch=es_ok,
cache=cache_ok,
)The status field answers “should traffic come here?” and only the
database gets a vote. The booleans answer “what is broken?” for a human.
That split matters more than it looks. If Redis being down marked instances unhealthy, a cache outage would pull the entire fleet out of rotation at once — converting a slow site into no site, because of a component that was supposed to be optional. A dependency in your health check is a dependency that can take you down.
Liveness and readiness are different questions
LIVENESS "is this process wedged?" failing -> RESTART it
READINESS "can it serve traffic now?" failing -> stop sending, don't restartConflating them causes a specific and nasty failure. If a readiness failure triggers restarts, then a database outage — which makes every instance unready — restarts your entire fleet in a loop, and the fleet is now also cold and hammering the database with reconnections while it recovers.
Readiness should also fail during startup, before caches are warm and connections are established, so an instance does not receive traffic it cannot yet serve.
Stateless, and what it actually costs
Everything above assumes any server can handle any request. That assumption is the real work.
The classic violation is server-side sessions. A user logs in, the server stores the session in memory, and now only that server knows who they are:
login ──> [ app 1 ] sessions = {abc123: user 7}
next ──> [ app 2 ] sessions = {} -> "please log in"There are three ways out, and only two are good.
Sticky sessions pin each user to one server. It is the tempting fix and it is a trap: you cannot deploy without logging people out, one server dying logs out everyone on it, load becomes uneven and stays uneven, and autoscaling cannot help because new servers get no traffic.
A shared session store — sessions in Redis rather than in memory — works properly. Any server can serve any request. The cost is a network round trip per request and a new critical dependency, which brings back the availability question above.
A signed token puts the session in the request. The server verifies a signature and needs no lookup at all. This is what StayHub does:
def create_access_token(user: User) -> str:
now = datetime.now(UTC)
expires = now + timedelta(minutes=settings.access_token_expire_minutes)
claims = {
"sub": str(user.public_id),
"email": user.email,
"role": user.role.value,
"iat": int(now.timestamp()),
"exp": int(expires.timestamp()),Any instance with the shared secret can verify that token. Nothing is stored, so nothing has to be shared, and a server can be destroyed mid-session with no consequence.
The cost is real and specific: you cannot revoke a token before it expires, because nothing is consulted. Ban a user and their token keeps working until it lapses. The usual mitigations are short lifetimes plus refresh tokens, or a revocation list — which reintroduces the lookup you were avoiding, though only for the rare case.
StayHub takes a middle path: it re-reads the user row on every request anyway.
user = db.execute(
select(User).where(User.public_id == claims["sub"], User.deleted.is_(False))
).scalar_one_or_none()
if user is None:
raise UnauthorizedException("Your account is no longer active.")
return userThe token proves who; the database says what they currently are. A deleted or demoted account stops working immediately, and the lookup is a primary-key hit that the database serves from memory.
The other state people forget
Sessions are the obvious one. These are the ones that ship to production and then break:
- Uploaded files written to local disk. The next request lands on another server and the file is not there. Object storage, not a filesystem.
- In-process caches. Not fatal, but each server has its own, so invalidation on one leaves the others stale — and users see values flip as they are balanced around.
- Scheduled jobs in the app process. Ten servers means the nightly billing run executes ten times. This one is expensive.
- Rate limit counters in memory. A limit of 100 becomes an effective limit of 100 × the number of servers. The rate limiting post is about exactly this.
- WebSocket connections. Genuinely stateful and cannot be made otherwise — the connection lives on one machine. The answer is a registry of which server holds which user, which is why chat systems have a component nothing else does.
Deploying without dropping requests
Once traffic is balanced across a pool, you can replace instances one at a time. Doing it without errors takes one more step than people expect.
1. mark instance NOT READY LB stops sending NEW requests
2. wait for in-flight to finish ~30s "connection draining"
3. SIGTERM the process it finishes and exits cleanly
4. start the replacement
5. wait for it to pass readiness NOT liveness — it may need warming
6. put it back in rotation
7. next instanceStep 2 is the one that gets skipped, and skipping it produces a small, constant trickle of 502s during every deploy that nobody can reproduce afterwards. A request already being served when the process dies is a failed request; draining is simply waiting for those to finish.
Step 5 matters for a different reason. A freshly started instance has an empty connection pool and a cold cache, so its first requests are slow. Sending it a full share immediately means a latency spike on every deploy.
The same mechanism gives you canary deploys: weight the new version to 5% of traffic, watch its error rate against the others, then shift the rest or roll back. That is a capability you get free from having a load balancer, and it is worth mentioning as a benefit alongside throughput.
Autoscaling, and the trap in it
Since the tier is stateless, machines can be added and removed automatically — usually on CPU, request rate, or a queue depth.
The trap is that scaling up is not instant. Provisioning a machine, starting the runtime, warming connections and passing readiness takes two to five minutes, and traffic spikes take seconds. An autoscaler alone does not protect you from a sudden spike; it protects you from a sustained one.
traffic ▁▁▁▁▁▁▇▇▇▇▇▇▇▇▇▇▇▇▇▇
capacity ▁▁▁▁▁▁▁▁▁▁▁▃▃▃▃▇▇▇▇▇
^^^^^
2-5 minutes of being overloaded
while the new instances bootSo the answers are: keep headroom rather than running at 80% CPU; scale on a leading indicator (queue depth, latency) rather than a lagging one (CPU); pre-scale for events you know about; and set a sensible floor so you are never at one instance.
The other trap is flapping — scaling down as soon as load drops, then back up thirty seconds later. Cooldown periods and asymmetric thresholds (scale up eagerly, down reluctantly) are the usual fix.
The load balancer is now the single point of failure
Worth saying, because it is the obvious follow-up question. You removed the SPOF from the app tier and created one in front of it.
DNS returns several IPs (round robin)
│
┌───────┴───────┐
v v
[ LB A ] [ LB B ] active/active, different zones
└───────┬───────┘
v
[ app 1..N ]Two answers in practice. Run a pair, sharing a floating IP that moves on failure. Or use a managed load balancer, which is already several machines across several availability zones pretending to be one thing — and this is the honest reason most teams use a cloud balancer.
DNS-level round robin sits above both. It is crude — clients cache DNS answers and will keep using a dead address until the TTL expires — so it distributes but does not fail over quickly. It is worth knowing precisely because of that weakness: a short TTL is the difference between a two-minute recovery and an hour of some users hitting a dead address, and TTLs are routinely left at their defaults.
There is a nice symmetry to notice here. Every layer of this stack solves the availability problem the same way — by having more than one of the thing and a way to detect which ones are broken. DNS has several A records, the balancer pair has a floating IP, the app tier has health checks, and further down the database has a standby. The mechanism differs; the shape does not.
Getting users to the nearest region
Everything so far is one datacenter. Serving users on several continents adds a routing layer above the load balancer, and there are two mechanisms.
GeoDNS answers the same hostname with different IPs depending on where the query came from — European users get the Frankfurt balancer, American users get Virginia. It is simple and it inherits DNS’s weakness: answers are cached, so failing a region over is as slow as the TTL.
Anycast announces the same IP address from many locations and lets internet routing deliver each packet to the nearest one. Failover is nearly immediate, because it happens at the network layer rather than by clients re-resolving. It is what CDNs use.
Both raise a question they cannot answer: your database is in one region. Serving a European read from Frankfurt is only useful if the data is there too, which means read replicas per region — and now every one of them lags. Multi-region is not a load balancing decision, it is a data decision, and it is genuinely hard. Reaching for it before you need it is one of the more expensive mistakes available.
Not sending the request at all
The cheapest request is the one that never arrives. Before scaling the fleet, remove work from it.
browser ──> [ CDN edge, near the user ]
│ hit -> ~10ms, your servers never knew
│ miss
v
[ LB ] ──> [ app ] ──> [ origin / object store ]A CDN is a global network of caches near users. It answers for anything static — images, CSS, JavaScript, fonts, video — and for a media-heavy site that is the overwhelming majority of requests by count and almost all of them by bytes.
The physics argument from the estimation post applies here and cannot be engineered around: a round trip from California to Europe is about 150ms no matter how fast your servers are. Only moving the bytes closer helps.
Cache lifetime is controlled by headers, and the pattern that makes it safe is fingerprinting:
/static/app.a3f9c2.js Cache-Control: max-age=31536000, immutable
^^^^^^ content hash — a new build is a NEW URL,
so a year-long cache is never stale
/index.html Cache-Control: no-cache
must revalidate — it names the fingerprinted filesFingerprinted assets cache forever safely, because changing the content changes the URL. The small HTML file that points at them is the only thing that must be revalidated. This site is built exactly this way.
Protecting the servers behind it
A load balancer distributes load. It does not, by itself, protect anything — and the point where all traffic converges is the natural place to add protection, because it is the only place that sees everything.
Three mechanisms belong here, and they are worth naming because interviewers ask what happens when traffic is not merely high but hostile or pathological.
Rate limiting at the edge. A request refused at the balancer costs no worker, no database connection and no log line. A request refused inside your application has already cost all three. Fine-grained per-user limits still belong in the application — that is where you know who is calling — but a coarse per-IP limit belongs out here.
Timeouts, everywhere. The default in most clients is no timeout at all, which means “wait forever”. One slow dependency then occupies every worker in the fleet, and a partial failure becomes a total one:
no timeout: 200 workers x waiting forever on a hung service
= 0 workers left for the 95% of requests that
do not touch that service at all
with timeout: those requests fail in 2s and free the workerEvery timeout in a chain should also be shorter than the one above it. If the balancer gives up after 30 seconds and your database query has no limit, the client is long gone while the query grinds on holding a connection.
Circuit breakers. After N consecutive failures against a dependency, stop calling it for a while and fail immediately instead. This is counter-intuitive — you are choosing to fail requests that might have worked — and it is right, because retrying a struggling service is how you keep it struggling. The retries are the load.
The same logic explains why retries need jitter. If every client retries after exactly one second, the recovering service is hit by a synchronised wave, fails again, and the wave repeats. A random component spreads them out. The queues post makes the same argument about backoff.
Putting it together
client
│
├── static ──> [ CDN ] ──────────────> [ object store ]
│ (never touches an app server)
└── API ────> [ DNS ] ──> [ LB pair ]
│ layer 7, TLS terminated here
│ least connections
│ GET /health every 5s
┌──────────┼──────────┐
v v v
[ app 1 ] [ app 2 ] [ app 3 ] stateless:
│ │ │ - JWT, no sessions
└──────────┴──────────┘ - uploads -> object store
│ - jobs -> a worker
v - counters -> Redis
[ database ]What to say when asked
- Layer 7 for an HTTP API, because path routing and real health checks are worth more than layer 4’s throughput at any scale you are likely to design for.
- Least connections when endpoint costs vary, round robin when they do not.
- Health checks report dependencies separately, and only the ones you truly cannot serve without get a vote on whether traffic arrives.
- Liveness restarts, readiness diverts. Never let the second trigger the first.
- Stateless via signed tokens, with the cost named out loud: revocation. Not sticky sessions.
- Run the balancer in pairs, or use a managed one, and say why.
- CDN first. Removing requests beats distributing them.
The through-line: the load balancer itself is the simple part. Everything hard about this topic is a consequence of running more than one of something — state has nowhere to live, failures have to be detected rather than noticed, and deploys become a rolling operation instead of a restart. Get those right and the balancer is a config file.
Next: caching — the other way to stop work reaching the parts of your system that cannot scale horizontally.