Debugging a Cloudflare 521 error on a Ghost blog

My Ghost blog started returning a Cloudflare Error 521: Web server is down. This is a note on how I diagnosed and fixed it.

What 521 means

A 521 comes from Cloudflare, not from the site itself. Cloudflare is working, but it cannot open a connection to the origin server behind it. So the problem is on the server, not in DNS or Cloudflare.

Step 1: Check server resources

A full disk or exhausted memory is a common cause, so I checked those first:

df -h      # disk
free -m    # memory

Disk was at 17% and memory had plenty free. The server was running fine, so the cause was a specific service rather than the machine.

Step 2: Find which service is down

Cloudflare depends on three services here: nginx (reverse proxy), Ghost, and MySQL (database). I checked nginx and MySQL:

sudo systemctl status nginx --no-pager
sudo systemctl status mysql --no-pager

MySQL was active (running). nginx was failed, and it had been failed for 6 days.

Cloudflare connects to nginx first. If nginx is down, every request returns a 521 regardless of the state of Ghost or the database.

Step 3: Find why nginx failed

The config test passed but the start failed. The logs showed the reason:

sudo journalctl -u nginx --no-pager -n 30

[emerg] host not found in upstream "ap.ghost.org"
in /etc/nginx/sites-enabled/kinakomemo.com-ssl.conf:24

The nginx config has an upstream pointing at ap.ghost.org, which is Ghost's ActivityPub integration. nginx resolves all upstream hostnames at startup, and if it cannot resolve even one, it refuses to start.

nginx had restarted cleanly for weeks. During a restart on July 31, DNS resolution for that hostname failed, so nginx did not start and stayed down until I looked into it.

The fix

Confirm the hostname resolves again, run the config test, then start nginx:

getent hosts ap.ghost.org    # should return an IP
sudo nginx -t                # full config test, including upstreams
sudo systemctl start nginx
sudo systemctl status nginx --no-pager

Once nginx is active (running), the 521 clears after Cloudflare reconnects to the origin.

Preventing a repeat

Because nginx resolves upstream hostnames at startup, a single DNS failure for ap.ghost.org can stop the whole server from serving. To avoid depending on startup-time resolution, use a resolver directive with a variable in proxy_pass so nginx resolves the hostname per request and starts even when the upstream is temporarily unreachable.