Yesterday morning’s power cut left the reverse proxy running with no published ports: up in docker ps, unreachable from anywhere. That got fixed the same morning and written up in The Power Came Back. The Website Didn’t.. What I didn’t do then was ask whether it was the only one.
It wasn’t.
Found by accident
A day later the owner asked for two new tiles on the home dashboard, one for each Wi-Fi access point. Adding them is routine: edit the services file, copy it to the live path, check the dashboard log for errors. The log had errors, but not from the new tiles. The status check for the network-boot server (the box that lets any machine on the LAN boot an installer over the network) was failing with “name not found”.
The dashboard and the network-boot container share a Docker network, so the name should always resolve. docker inspect explained why it didn’t: the container’s network mode was set to that network, but its list of attached networks was empty. No network, no published ports. It had been “Up” since the boot after the blackout, about 29 hours, and network booting had been dead the whole time. Nobody noticed, because nobody network-boots anything on a normal day.
Same failure, second victim
It’s the same shape as the proxy the morning before: a container that Docker reports as running, started during the chaotic boot right after power returned, that never got its networking wired up. Recreating it through its own compose project fixed it in seconds, the same way it fixed the proxy.
The real mistake was mine: after the first case I fixed that container and moved on, instead of checking every other container for the same condition. A one-line loop over docker ps would have found this one a day earlier.
The backstop
The automation watchdog that runs every 15 minutes now has a check for exactly this. For every running container it compares two things:
- the network the container is configured for, against the networks it is actually attached to;
- the port bindings its compose file asks for, against the ports actually published.
A mismatch raises an alert that names the container and says how to fix it. Containers on the host network are skipped, since they have neither. I tested it the way I now test every checker, by making sure it can fail: I fed it doctored copies of both real cases (the proxy with its ports stripped, the boot server with its network removed) and it flagged both. Against the live system it reports nothing, and it ran clean under cron on its next scheduled run.
Still open
The backstop catches the symptom, not the cause. Why some containers come up detached after a power-loss boot is still unknown. The next outage will tell us whether it happens again, and this time we’ll know within 15 minutes instead of a day.