Building a Zero-Downtime Load Balancer: Active-Passive HA with nginx, keepalived, and Go

TL;DR: I built a full active-passive HA load balancer — two nodes with nginx, a Go backend, keepalived/VRRP, Docker Compose, a floating virtual IP, observability stack, and an automated test suite that asserts failover works. Not "configured" — built and tested. This post covers the five design decisions that actually matter, how failure detection is layered across two independent planes, the measured failover budget, and why "high availability" is just a rumor if CI doesn't verify it. Repo: 2SSK/nginx-load-balancer-lab.

Requirements

• Active-passive HA at the load-balancer tier. The load balancer is the network's single point of failure. It must itself be redundant, with a shared identity clients don't have to change during failover.
• Reproducible with one command. Image builds, TLS certs, config templates, the whole topology — ./scripts/deploy.sh up brings up everything.
• Testable. Failover must be assertable by script, runnable in CI.
• Observable. Metrics, logs, and alerting must make the system's behavior visible, not inferred.
• Security-hardened by default. Least privilege, secrets injected at runtime, monitoring isolated from production traffic.

The Five Design Decisions That Matter

Most HA tutorials present config as a sequence of steps. I read configs as encoded decisions — each one a trade-off with alternatives that were considered and rejected. Here are the five decisions that define this system.

1. Active-passive, not active-active

Two load balancers could share traffic (active-active, "two-way real traffic" in VRRP terms), or one could carry traffic while the other stands ready (active-passive).

• Active-active doubles capacity and gives true load sharing, but requires both nodes to be genuinely symmetric, doubles the failure surface, and complicates stateful sessions.
• Active-passive caps capacity at a single node but makes failover semantics simple and predictable: one node owns the VIP, the standby is identical and warm.

For a front-tier load balancer, active-passive is the right call: capacity isn't the constraint — continuity is. And the pattern transfers directly: it's the on-prem analogue of an AWS ALB in a multi-AZ deployment with a health-checked target group, or an Azure Load Balancer with a backend pool. The cloud version replaces VRRP with a control plane, but the abstraction — "clients address one thing; the platform decides which node answers" — is identical.

Decision: active-passive, differences encoded as two numbers — priority 101 for the primary, 100 for the standby. Identical otherwise.

2. The virtual IP is the contract

Clients need a stable address that survives node failure. That's the virtual IP (VIP): an ordinary address on a network interface that keepalived adds and removes at runtime. On the primary, ip addr show shows the node's own address plus the VIP; on the standby, the VIP is simply not bound. The whole state difference between "primary" and "standby" is one interface address.

Failover is a physical relocation:

1. The primary stops advertising (or dies).
2. The standby promotes and binds the VIP to its interface.
3. It broadcasts a gratuitous ARP, so every switch and host on the L2 segment immediately updates its cached MAC mapping for that IP — no waiting for ARP cache timeout, no DNS change, no client reconfiguration.

Two implementation details on the VIP itself that tripped me up:

• Address placement. The VIP (172.28.0.200/24) lives inside the Docker bridge's /16 but outside the IPAM auto-assign range (.2–.254). Collision with a container address is structurally impossible, and Docker routes the whole /16 to the bridge, so the host needs zero custom routes. This is the kind of decision that looks trivial and prevents an entire class of intermittent production bugs.
• Unicast VRRP, not multicast. VRRP's default multicast announcements don't reliably traverse Docker bridge networks — and plenty of cloud/virtualized networks filter multicast too. A point-to-point unicast peering between known addresses works everywhere.

Decision: one stable address, movable at the network layer, with ARP-level propagation speed.

3. The node health model: one container, three processes

Each "server" runs three processes — nginx, the Go backend, and keepalived — supervised by a single PID 1 in one container. This looks unusual; it's deliberate.

The question keepalived answers is "is this node fit to serve?" That fitness depends on all three processes. Splitting them into separate containers would decouple the availability decision from the thing it protects — a node whose nginx died could keep the VIP while serving nothing. The supervisor uses wait -n, so if keepalived itself dies, the container exits and Docker restarts it: a half-alive load balancer is worse than a dead one, because a dead one triggers failover and a zombie one silently degrades service.

Here's the subtlety that took me a while to get right: killing the container immediately when nginx or the Go app crashes would actually be wrong. It would tear keepalived down before its own health script (chk_nginx) gets the chance to detect the failure and demote the node over VRRP — which is the whole point of running keepalived. So the supervisor waits on keepalived and a health watchdog; a failure in nginx or the app first flows through VRRP (fast failover, no container restart), and only if local health stays broken for ~60 seconds does the watchdog force a restart. Two recovery paths with deliberate, different latencies — not one blunt hammer.

Decision: availability state lives with the node, not the process; recovery is layered and time-budgeted.

4. Failure detection is layered across two planes

This is the load-bearing insight, so I'll state it plainly: this system has two independent failure-detection planes, because a single plane cannot detect the full failure spectrum.

The traffic plane (nginx). Every node runs an upstream pool containing both backends:

upstream backend {
    least_conn;
    server ubuntu-server-01:8080 max_fails=3 fail_timeout=10s;
    server ubuntu-server-02:8080 max_fails=3 fail_timeout=10s;
    keepalive 32;
}

This detects backend failure and redistributes traffic across the pool — even on the node that holds the VIP. A dead or wedged backend gets excluded via passive failure detection (max_fails=3 in a fail_timeout=10s window), and proxy_next_upstream retries the same request on the next backend so the client sees, at most, one slow response before the pool self-heals.

The availability plane (keepalived/VRRP). This detects node failure and moves the VIP — the shared identity — to the node that can actually serve. Node fitness is tied to a tracked script:

vrrp_script chk_nginx {
    script "/usr/bin/curl -sfk --max-time 2 https://localhost/health"
    interval 2    # poll every 2s
    fall 2        # 2 consecutive failures → demote
    rise 2        # 2 consecutive successes → promote/restore
}

The two planes are independent and both required:

Any design that relies on one plane leaves part of the failure spectrum unhandled — typically the hang, which is the realistic outage. Crashes are easy. Hangs are what actually take down production.

5. Timeouts are a budget, not a tuning afterthought

Defaults are the enemy here. nginx's default backend timeouts are 60 seconds — meaning a hung backend ties up requests for a minute before anything counts as failure. I set proxy_connect_timeout 2s; proxy_send_timeout 3s; proxy_read_timeout 3s; to collapse that to ~2–3 seconds. And --max-time 2 on the health script guarantees that a hung service cannot hang the checker itself — the checker dies before the thing it's checking, which is the cardinal rule of health checks.

The reason this matters is the failover budget. Recovery time is not a number you guess; it's a sum of constants:

detection latency  = fall(2) × interval(2s)          ≈ 4s  (worst case, health-driven)
                   + VRRP election + ARP propagation  ≈ 1–2s

So the measured failover for health-driven scenarios lands in the ~5–7 second range — and the test suite verifies that number end-to-end rather than assuming it. Hard kill of the primary detects via VRRP advertisement silence (1s interval + election), which is why that scenario is faster. When someone asks "how fast is your failover?", the answer should be a balanced equation, not an anecdote.

Three More Decisions That Complete the Failure Model

Crash vs hang

A crash fails fast — the OS refuses the connection instantly. A hang (process alive, TCP accepted, never replies — deadlock, thread exhaustion, slow downstream) is the realistic outage, and it's what the timeout budget exists for. The Go backend can simulate both on command (/hang/on for the hang, /crash for the exit, /health/fail for the explicit unhealthy state), so every detection path is exercised for real rather than inspected for correctness.

Health is a question with a subject

The health check must ask "is THIS node fit", not "is some node in the pool fit". If chk_nginx hits a location that passes through the upstream pool, it can be answered by the peer's healthy backend — and a node that serves nothing but errors stays "healthy" forever. The system uses dedicated /health and /ready locations that bypass the pool and hit localhost directly:

location = /health {
    proxy_pass http://localhost:8080/health;
}

The /health vs /ready split is preserved from Kubernetes semantics: liveness ("should traffic route here now") vs readiness ("has this instance finished starting"). They diverge legitimately in production (ready but overloaded = healthy, but not ready for new work); keeping them separate from the start is cheaper than retrofitting the distinction later.

Identity is runtime configuration

One image builds both nodes; KEEPALIVED_STATE, KEEPALIVED_PRIORITY, and PEER_HOST env vars are substituted into a config template at startup. Same for the backend's HOSTNAME/SERVER_COLOR/VERSION. No dual-maintained node-specific configs, no image per role. The startup script validates both the keepalived state enum and the priority before substitution, and detects the VRRP interface by matching the VIP's subnet prefix — Docker does not guarantee interface ordering, and hardcoding eth0 is how you bind VRRP to the wrong network.

Verification: Availability as a Tested Property

The test suite (scripts/test-ha.sh) is the system's most important artifact, because it's what turns "high availability" from a claim into a property.

The philosophy: assert the contract, not the activity. The tests don't check "did the script run" — they assert:

1. The VIP is bound to exactly one node (checked inside each container with ip addr show).
2. Both backends report healthy.
3. The VIP itself answers HTTPS — not the host-published port.

Point 3 is a correctness detail I almost got wrong. The host port 8443 belongs to server-01 only; kill nginx there and localhost:8443 stops answering even though the VIP has correctly migrated to the standby. Probing the host port tests an implementation detail; probing https://172.28.0.200/health tests the contract clients depend on. This is the same trap that bites production DR drills: you test by poking a specific node and then have to explain why the system that "failed" the drill was actually fine.

Four scenarios, escalating in realism:

Every scenario ends with recovery — restore health, un-hang, recreate — followed by re-asserting the cluster returned to a good state. A failover test that verifies failover but not fail-back is half a test. Suite result: Passed: 10/10; it runs in CI alongside the build, so an availability regression is a failed pipeline, not a future incident.

Observability: Probes Follow the Contract

The monitoring stack (Prometheus, Grafana, Loki, Alertmanager, exporters — 11 services) exists to keep the loop closed: alert → runbook → action. The design choices worth documenting:

• Probe the VIP, not the nodes. Blackbox-exporter checks the virtual address, mirroring the test suite's assert_vip_reachable. Probing nodes proves components; probing the VIP proves the contract clients actually use.
• Three metric layers, deliberately. Application metrics (request rate, latency, errors), infrastructure metrics (CPU, memory, disk), and load-balancer metrics (connection counts, upstream distribution). Real incidents almost always require correlating two layers; you don't get to choose which two in advance.
• Observability caught a real design bug. The Grafana dashboard fetches cluster state through the VIP, so its requests are themselves load-balanced — the card labeled "self" kept answering from alternating backends and swapping identities every couple of seconds in the live view. Root cause: the app's own status endpoint had a self-vs-peer ambiguity under load-balanced access. The fix — key dashboard cards by hostname, and target chaos actions deterministically (/cluster/target/hang/on?host=X instead of "the peer") — is exactly the kind of subtle distributed-systems correctness issue you'd want caught before production, not after.
• Security posture throughout. The entire observability plane is bound to 127.0.0.1 (reachable via SSH tunnel only); Alertmanager SMTP credentials are injected at runtime via awk template substitution (the official image has no envsubst, and naive sed substitution turns credential characters into injection surface); metrics endpoints are ACL'd to management networks; the /crash endpoint is gated behind a CRASH_TOKEN so failure injection is deliberate, not accidental.

Operational Maturity

I've learned the hard way that architecture without operational tooling is just a diagram. The repo carries the artifacts:

• Single-command lifecycle — deploy.sh for up/down/build/test/status/logs, with a rolling-restart order that is itself a decision: standby first, then primary, so the standby takes the VIP before the primary returns to reclaim it (preemption).
• Config validation before runtime — nginx -t and keepalived -t run on every deploy, and again in the supervisor before processes start (fail fast rather than boot a broken config).
• Container hardening — cap_drop: ALL with an explicit allowlist (NET_ADMIN, NET_RAW, NET_BIND_SERVICE, …), nginx server_tokens off, TLS 1.2/1.3 with a Mozilla-intermediate suite, security headers, rate limiting, Slowloris protection, HTTP→HTTPS except where exporters intentionally scrape the HTTP listener.
• Runbooks for the operational states — incident response, manual failover, recovery, cert renewal, container recovery — so the response to an alert is a documented procedure, not tribal memory.
• Documentation as code — README architecture, from-first-principles concept doc, and a ROADMAP reflecting the actual build order. I'll be honest: there's drift in the repo's older docs versus the live config. Documentation that disagrees with deployed reality is untrustworthy documentation — lesson applied to myself.

What I'd Tell You to Take Away

Strip away nginx and keepalived and this lab is really about engineering principles I'd carry into any platform:

1. Recovery time is a budget you compute, not a guess. Every constant in the system — check interval, fall, timeout values — feeds an actual measured number for how fast the system detects and recovers from each failure class.
2. Failure is a spectrum, so detection is layered. Crash, hang, and degradation are different events requiring different planes. One health check cannot see all of them.
3. Health checks must have a clear subject. Liveness and readiness are distinct questions, and "is this node fit" is never "is the pool reaching a healthy instance."
4. Availability claims require assertions in CI. If it isn't verified by a failing pipeline, it's a rumor. Runbooks and dashboards are the operational layer of the same discipline.
5. Chaos is an API, not an accident. Failure injection is deliberate, scoped, token-gated, and observable — the difference between engineering resilience and just hoping to survive it.
6. One identity, many roles. Same image, same config template, different env — reproducibility as a configuration principle, applicable to containers, Terraform modules, and Helm charts alike.

What This Doesn't Solve

In the interest of calibrated claims: this lab proves the discipline on a Docker bridge, not the entire problem in a datacenter. Real deployments face switch ARP behavior, VRRP-unaware middleboxes, and network partitions this environment never exercises. NET_ADMIN is a real privilege that deserves production scrutiny. Two nodes remain a minimum, not a recommendation — shared fate between a pair is still fate-sharing. But the decision framework, the failure taxonomy, the verification strategy, and the operational artifacts are the parts that transfer.

The complete system — Docker Compose topology, nginx and keepalived configs, failure-injection Go backend, test suite, runbooks, CI — is open source: github.com/2SSK/nginx-load-balancer-lab

Related posts from the same operating theme: a PostgreSQL 17 migration postmortem (data-plane HA, including WAL replication and what breaks it) and systemd production service management (supervision semantics outside containers).

Find me on GitHub — questions, corrections, and DR-drill war stories welcome.