graph TD A[Unattended install config] --> B[Throwaway VM] B -->|run 1| C[Web server died<br/>when SSH closed] B -->|run 2| D[ISO didn't fit<br/>in RAM] B -->|run 3| E[YAML read a command<br/>as a key/value pair] B -->|run 4| F[VM disk hid<br/>its serial number] B -->|run 5| G[Clean install<br/>in 7 minutes] G --> H[Ready for the real machine]
The main server is going back to bare metal. Two months ago it moved into a virtual machine on a hypervisor, mostly for one feature: snapshots before risky changes. That feature turned out not to work with the way the data disks are attached. Meanwhile the extra layer brought a network-card bug, a boot failure after a disk swap, and a second machine to keep alive just to host the first. So the plan is to wipe the hypervisor’s disk and install plain Ubuntu directly on the hardware.
The catch: the owner wanted to do it remotely. This box has no remote console. It’s a desktop-class machine without Intel’s management engine features (we checked: the firmware reports no AMT at all), so if the install hangs, nobody can see the screen.
That makes the install config the single most important file in the whole job. It has to answer every installer question correctly, pick the right disk out of four, come up on the network, and let us in. And it has to do that the first time, on the real machine.
So we didn’t try it first on the real machine.
The test bench
The hypervisor was still there, so it hosted its own replacement’s dress rehearsal: a throwaway VM, booted exactly the way the real machine would boot: same installer kernel, same command line, same config, same web server handing out the ISO. The only differences were the disk serial number, the network card’s address and the IP. If the config could install that VM unattended, it could install the real thing.
It failed four times before it worked. Every failure would have been worse on real hardware.
Run 1: nothing happened. The installer booted, asked the web server for the ISO, and got “connection refused”. The little web server serving the files had been started as a per-user background service, and that service quietly stopped the moment the SSH session that started it closed. On the real day, that would have been a machine sitting at an error screen with nobody watching. Fix: start it fully detached.
Run 2: “No space left on device” at 72%. The installer downloads the whole ISO into a RAM disk sized at half the machine’s memory. The test VM had 6 GB, so the RAM disk was 3 GB, and the ISO is 2.9 GB plus everything else. This one only bit the test (the real machine has 64 GB), but it’s the kind of thing worth knowing before the day, not during it.
Run 3: “Malformed autoinstall in ‘late-commands’“. This was a real bug in the config. One of the post-install commands writes a sudo rule, and that rule contains NOPASSWD: ALL. In YAML, a colon followed by a space inside an unquoted string means “this is a key and a value”. So the line wasn’t a command at all; it was a tiny dictionary. The render script had checked that the file was valid YAML, which it was. It hadn’t checked that the YAML meant what we thought.
The good news hiding in this one: the installer validates the whole config before touching any disk. The same mistake on the real machine would have stopped at a prompt, not halfway through a wipe. The render script now checks that every command is actually a string, and we confirmed it rejects the broken file with the same complaint the installer gave.
Run 4: “Failed to find matching device”. The config picks the install disk by serial number, on purpose: the machine has three data disks full of photos and media, and the one thing this install must never do is guess. In the VM, it found nothing. Opening a shell on the stuck installer showed why: the VM platform puts the virtual disk’s serial in a different property than the one the installer matches against. The real NVMe drive reports it where the installer looks (we checked on the hardware). So this was a test-bench quirk. But it also proved the safety net: no match means stop, nothing touched.
Run 5: clean. About seven minutes from power-on to a powered-off, installed system. Booting it showed the right hostname, the right user, passwordless sudo, one ext4 partition, a static address, SSH keys working, password login off, and logs that persist from the first boot.
One more safety net
The scariest remote failure is an installer stuck at an error prompt with nobody there to read it. So the config’s very first step now drops our SSH keys into the installer’s own environment, not just the installed system. On a rerun we SSHed into the live installer mid-install from two different machines, read its logs, and watched it work. If the real install stalls, we can see why and reboot it from here.
Why not just use GRUB’s “boot this once”?
The obvious way to boot an installer one time is GRUB’s one-shot entry. It doesn’t work here: GRUB records “already used” in a small file on the boot disk, and on this machine that file lives on a volume type GRUB can read but not write. The one-shot would never clear, so a failed install would just boot the installer again, forever.
Instead, the installer kernel sits on the EFI partition with a firmware boot entry that isn’t in the normal boot order, and the firmware’s own BootNext setting points at it for exactly one boot. The firmware clears that itself. If the kernel doesn’t start for any reason, the firmware moves on to the normal boot order, which brings back the old system, which restarts the server VM on its own.
What we took away
- A test that can’t fail isn’t a test. Each of these four would have shown up on the real day instead. Four restarts of a throwaway VM cost about an hour.
- “Valid” isn’t “correct”. The YAML parsed fine. It just didn’t mean what we wrote.
- Prefer the failure that stops early. Matching the disk by serial, validating before writing, and a one-time boot that falls back: each one makes the likely failure “nothing happened” instead of “something half-happened”.
The real install hasn’t happened yet. When it does, it’ll be the sixth run of a config that has already failed every way we could think of making it fail.