At about 05:10 local time the power went out. The UPS did exactly what it was bought for: about seven minutes on battery, a low-battery signal, then the secondary server shut down first and the main one after it, cleanly. I have watched that chain work in a staged test and in two real cuts since, and it keeps working. That part of the story is boring, which is the best kind.
The interesting part started when the power returned.
Three short boots
The mains flapped for about twenty-five minutes. The main server booted, got shut down again by the UPS logic, booted again, and finally stayed up. Every boot is a small race: the network link, the container runtime and the services all come up in whatever order the clock happens to give them.
Everything said “Up”
By the time I looked, the container list was green. Fifty-four containers running, none unhealthy. And yet every internal site, and the blog, refused connections.
The culprit was the reverse proxy. Its container was running, but it had no published ports. The proxy binds to one specific LAN address on purpose, so it never exposes HTTPS on the Docker bridge. On this boot the network link came up about five seconds after the container runtime had already started the containers. The address wasn’t there yet when Docker tried to publish the ports, so it silently didn’t, and nothing ever retried.
A plain docker restart didn’t help, because a restart doesn’t redo the port publishing that a create does. Recreating the container through its compose project did. A second service, the login gate, had been restart-looping the whole time, because its startup check goes through the very proxy that wasn’t listening. It recovered once I gave it a push.
The lesson is the one I keep relearning: “running” is not “reachable”. My boot checks look at container status. Nothing looks at whether the ports are actually open on the address that matters.
The backup that died halfway
The nightly backup starts at 03:00 UTC. The power failed eleven minutes into it. The local copy had finished, but the offsite upload was killed mid-run by the shutdown, and the next two steps never ran at all.
That mattered because the night before, 27 digitised family videos (22 GiB) had been added to the photo library. Until the offsite copy completed, they existed only on the home machines. So before doing anything else with those files, I re-ran the whole backup by hand and then checked each destination: one directly by searching its newest snapshot for a new file, the other two from the script’s own log. (My first attempt didn’t even start. The log redirect ran as my user, not as root, and was refused. A command that exits quietly isn’t the same as a command that ran.)
The second server stayed off
The standby server never came back by itself. I sent it a Wake-on-LAN packet and got nothing, which is common: a machine that lost power completely often can’t be woken that way. The power button worked.
What I’m doing about it
- The router, switch and access points are not on the UPS, so the whole network went dark with the power, including DNS. A small UPS for those is now the next purchase, with the router set to outlast the servers.
- The “ports didn’t publish” case still has no alert, and that gap is on my list.
On the plus side, the VHS videos are now shared with the family through a read-only, password-protected link that expires in a month. That part, at least, worked on the first try from a phone on mobile data.