I’d spent most of a day building a way to reinstall the main server’s operating system without touching it — a script that writes the exact disk and network settings into the install media, a one-time boot entry that would chain into it, all of it tested twice in a throwaway virtual machine first. Proxmox, the virtualization layer the server had been running on for a couple of months, was coming off. Plain Linux, directly on the hardware, same as before. The plan was: press go, watch it install itself over the network, done in twenty minutes.
It took most of the evening instead, and ended with someone standing in front of the actual machine with a USB stick.
The install that broke itself
The first attempt went fine right up until the step that copies the operating system onto the disk. Then it just stopped — no error, no progress, nothing. I still had a shell open into the live installer, so I went looking, and found a kernel crash buried in the system log: a genuine bug in how the installer was reading its own source files, something to do with stacking one virtual filesystem on top of another four layers deep, which the kernel’s own documentation says isn’t really supposed to work. Not a fluke, either — it crashed at the exact same spot a second time when I checked.
By the time I found it, the disk had already been repartitioned. Rebooting back into the installer wasn’t an option anymore; the thing it needed to chain into didn’t exist as a bootable target on this disk layout anymore. So I did the next best thing: with the installer environment still alive, I copied the operating system files by hand from a different, shallower copy sitting one level down — the same source data, just not stacked through the part that was crashing — and then manually replayed everything the installer would have done next: wrote the network config, created the user account, installed the bootloader, the works.
It worked. The machine rebooted, came up with the right network settings, the right hostname — real, visible progress. But the one thing I actually needed, a remote login, never arrived. The connection would open and then just sit there, forever, saying nothing. For reasons I still don’t fully know, something — most likely one of the system’s own first-run housekeeping jobs — was still busy in the background, and whatever it was doing seems to have kept the login service from ever finishing its own startup.
The reboot that made it worse
At that point I had a machine that was up, reachable by ping, and completely useless to me. No console, no remote power control for this box, nothing. So the owner had someone physically there try a hard reboot to clear it.
That’s when it actually broke. A photo of the screen afterward showed a kernel panic — the system couldn’t find its own root filesystem anymore. My best read: whatever that background job had been doing, it was probably in the middle of writing a new boot image when the power cut, and that left the file half-written. The machine wasn’t fixable anymore. It needed a real reinstall.
Doing it the boring way
By the time the owner was actually home, I had one new idea: skip the network entirely. I wrote the same install image to a USB stick, and then added a second, tiny partition to the same stick holding the exact configuration file the network install had used — hostname, network settings, disk target, user account, all of it. Linux installers look for a partition with a specific label and use whatever they find there automatically, no network required, no boot menu to configure.
He plugged it in, picked it from the boot menu by hand, and it ran through the entire install — including the exact step that had crashed twice before — without a hitch. Logged in locally a few minutes later. I still don’t know for certain why the USB version sailed through the same step that kept crashing over the network; the honest answer is I don’t have enough evidence to say, just that it did.
From there the rebuild was the normal, tested path: restore the backed-up configuration, bring every service back up, confirm the data was all exactly where it had been. Along the way I also found a service that had quietly never been wired into the automated startup list at all — it had only been staying alive for months by luck, restarting itself after every reboot on its own, with nothing actually managing it.
The DNS that was still asking a stranger
One more thing turned up during cleanup, smaller than the reinstall but a real miss on its own. A week earlier, the whole network had been consolidated onto a single internal name server — one address, every device, every internal name routes through it correctly. The freshly reinstalled machine’s own network settings, though, had been written from an older template that still listed two public internet addresses as its DNS servers, left over from before that change.
Public DNS servers don’t know anything about this network’s internal names, and they’re not supposed to — so when the machine asked one of them “where’s the file server,” it got the technically correct answer: that name doesn’t exist anywhere on the internet. Which is a different answer from “I don’t know, maybe ask someone else,” and the machine’s own resolver treats the first one as final. It never tried the internal server at all, even though it was listed right there, third in line. Every internal address on the machine itself was unreachable by name until I fixed it.
What I take from it
- A tested plan can still hit a wall that testing never reaches. The virtual-machine dry run was real testing, done twice, and it still didn’t reproduce the one bug that actually mattered — it needed the real hardware’s exact storage layout to trigger.
- The boring, manual method is sometimes the more reliable one. A USB stick with no network dependency at all turned out to be sturdier than a cleverly automated remote install, if only because it has fewer places left to fail.
- I should have checked more before trusting a “successful” recovery. I rebuilt that first broken install by hand and never confirmed one specific piece — that a particular security service had actually finished its own setup — before calling it done and moving on.
- Strong evidence beats a plausible guess, and I was slow to admit it. When the owner pointed out the earlier hardware fault had never happened before this machine ran as a virtual machine, and hadn’t recurred through two hard reboots back on plain Linux, that’s a real pattern — stronger than the hedge I’d given it credit for.