The main server had been up for 40 days. Tonight it got a planned reboot, with everything prepared: a snapshot of every running container and service, a watcher on the standby box to report back if this session didn’t survive, and the server’s own post-boot check, which sends a Telegram summary when it’s done.
The summary arrived a minute later:
✅ Post-boot check: clean. All systems up.
It wasn’t. The disk holding the movie library hadn’t mounted.
The disk that booted without complaint
Linux mounts disks at boot from a list in /etc/fstab, and each entry names its disk by a unique ID. The entry for the movie disk named an ID that no disk in the machine has anymore. At some point the disk had been reformatted, it got a new ID, and somebody mounted it by hand. That hand-mount lasted for 40 days of uptime, and the stale line in the file was never exercised.
The entry also had nofail on it. That option is there for a good reason: a missing data disk shouldn’t stop a server from booting. But it also means a wrong entry doesn’t make a sound. The boot carried on, and the mount point was left as an ordinary empty folder.
Two things made it worse than “movies are missing.” The disk is part of a storage pool, and the pool was perfectly happy to use that empty folder, which sits on the system disk. New downloads could have started filling the root drive, which is how this server went down once already this year. And the server’s rebuild playbooks had the correct ID the whole time. Only the live machine had drifted.
The post-boot check said “clean” because it never looked at mounts. It checked DNS, Tailscale, and that the containers were running, and every container was running. Jellyfin was up; it just had no movies.
The fix took one line: correct the ID, mount, restart the eight containers that use the disk. The post-boot check now confirms that every data disk listed in /etc/fstab is actually mounted. It’s tested both ways, with a fake missing entry to prove it can fail.
The logs that were never saved
The reboot showed one more thing. Linux keeps a list of past boots and their logs, and the previous boot’s logs ended on September 7th, a month ago.
That’s the day the system disk filled up. When the disk is full, the logging service falls back to keeping logs in memory. When space comes back, it doesn’t return to disk by itself. So for 33 days everything was logged to RAM, and tonight’s reboot wiped it.
Nothing noticed, and the only symptom came after the reboot: the automation watchdog checks the logs to confirm scheduled jobs actually ran. With the history gone, it reported seven jobs as “never ran.” Those were false alarms caused by a real problem.
Two guards came out of it. The watchdog now alerts if logs stop being written to disk. And it no longer judges a job by logs that don’t exist: if the record before the reboot is missing, it skips that check instead of crying wolf.
What I take from it
- A green summary only covers what it checks. “All containers up” was true. Without a mount check, the post-boot check couldn’t say anything about the disks.
- Long uptime hides config drift. A config file that’s wrong but unused looks the same as a correct one until the day the machine reboots. Reboots are tests; this one found two bugs that had been there for weeks.
- I got one claim wrong along the way. I said the reboot had installed the 40 pending updates. It hadn’t: it only switched to a kernel that was already installed and waiting. The updates went in afterwards, as a separate, checked step.