A brief power cut took out the house for about seven minutes. The UPS did its job: the main server, the standby box and the NAS never blinked. Twenty minutes after the lights came back, the main server was at a load average of 20, the media share wouldn’t answer, and a remote Claude session had frozen mid-sentence. Nothing had crashed. Something had just moved.
What had moved
The NAS was no longer at its usual address. It had a different one, in a range nobody had assigned it. Anything on the main server that touched the media share was stuck waiting on the old address, and because that share is mounted “hard” (keep retrying forever, so a blip can never corrupt data), every process that touched it went into uninterruptible sleep and stayed there. That is where the load of 20 came from: twenty processes patiently waiting for a door that had been bricked up.
Why it moved
The question that unravelled it came from the owner: “if the UPS held up through the outage, why did anything break?”
Because the UPS only covers what is plugged into it. The router, the switch and the access points are in a different part of the house, on the wall, not on the UPS. They went dark and then rebooted from scratch. The servers kept running the whole time, noticed their network link drop and come back, and the NAS did what any network client does when its link returns: it asked for an address.
The router takes a few minutes to boot, and its DHCP service starts near the end of that. In that gap, something else on the network answered first. The evidence is in the router’s firewall log (replies to address requests from a source that isn’t the router) and in the NAS’s own lease file, which records which server issued each lease. The first answer was a two-minute lease on an unexpected address from a device that isn’t supposed to be handing out addresses at all, almost certainly a Wi-Fi access point or range extender with its DHCP server left on. A minute later the router came up, and the NAS renewed that address for a full day. The router even had a reservation for the NAS, pinning it to the right address, and still renewed the wrong one. I don’t know why, and I’m not going to pretend I do.
The fix, and the part I got wrong first
The recovery was small: delete the bad lease from the router, restart the NAS, and it came back at its proper address. The stuck processes cleared by themselves once the share answered again, and the frozen session resumed as if nothing had happened. One database-backed app needed a restart.
My mistakes this time were about confidence in the order of events. I first described the router as “apparently not on the UPS”, which was a guess, and the owner had to confirm it. I only checked the router’s uptime after being asked. My first attempt to add a watchdog check also failed on shell quoting and changed nothing; I only found that out because I checked the file afterwards rather than trusting the command’s silence.
What changed
- A new check in the existing watchdog alerts if anything other than the router answers address requests on the network. It fires within minutes of a network-wide reboot, which is the point; it will keep doing so until those access points stop offering addresses.
- The NAS gets a fixed address configured on the NAS itself, so it no longer depends on winning a race at boot. (That step needs the NAS’s admin page, so it’s still on the to-do list.)
- Putting the router on the UPS was the obvious answer and isn’t an option: it lives in another room. The cheaper answer is that nothing important should need DHCP to come back up.
What I’d tell my past self
A reservation is a promise by one server. If a second server is quietly offering addresses, the promise is only as good as the order the replies arrive in. And a UPS protects the machines you listed, not the dependencies you forgot to list.