Recover or reinstall your XenServer PV VMs

This blog post will explain how to deal with your existing PV guests, when they can't boot anymore.

The problem

Typically, when you failed your installation of a PV guest, you got:

The bootloader for this VM returned an error -- did the VM installation succeed? Error from bootloader: no bootable disk

Or even:

The bootloader for this VM returned an error - did the VM installation succeed? Unable to find partition containing kernel 

It could also happened if your /boot is corrupted for any reason (IO error, removed grub files, etc.)

The previous solutions were:

• if it's after a failed installation, you are good to re-create a new VM
• if it occurs on a used VM, you'll need to make a lot of complicated stuff to access your data on your existing VDIs (detach the disk, attach it to another VM, or mount it inside the host...)

Side note

If you experience this error:

Internal error: xenopsd internal error: XenguestHelper.Xenctrl_dom_linux_build_failure(2, " panic: xc_dom_core.c:540: xc_dom_find_loader: no loader\\\"")

That's another story: it's because your XenServer is too old and it's not compatible with Grub2. XenServer 6.5 solves the issue, as explained in a previous article. If you don't want to upgrade, there is also another way to fix it.

The solution

When you build a software like Xen Orchestra, you need to understand how XenServer works under the hood. One of our customers asked to allow its clients to be able to reinstall their delegated VMs. Very trivial with a XenServer HVM guest: just change the boot order. But what about XenServer PV guests?

PV guests are special

If you just want the solution, go to the next paragraph.

PV guests are installed with the help of scripts executed on your XenServer host (Dom0).

Basically, during their first boot after be provisioned, they will be booted with a script called eliloader. This Python script will manage all the aspects of a PV installation, for each major distro. After this first boot, it will prepare the VM to boot normally, on the grub config inside the guest.

But if your installation fails, you are totally doomed for the next boot. Or if you bootloader is corrupted, you can't boot it anymore.

That's why we created a "Recovery mode". It exists for HVM guest since 4.10. But it's far from trivial to make the same feature for PV. Because to boot again on an ISO, you need to:

• set the PV_bootloader to eliloader
• put XAPI other_config.install-repository to cdrom
• give the CD drive a bootable flag to true
• remove the bootable flag on all other drives
• recover the install-distro from its parent template

All of this complicated stuff is done by just clicking on one button.

Recovery mode

Inside your VM view, just click on this button:

It will boot on the CD inside the CD drive. You can now:

• install a fresh OS on top of the previous one[^n]
• abort the installation to drop a shell, in order to recover your files, or reinstall Grub

All your partitions are easily accessible:

# fdisk -l

Disk /dev/xvde: 8589 MB

Device Boot      Start         End      Blocks   Id  System
/dev/xvde1            2048    15988735     7993344   83  Linux
/dev/xvde2        15990782    16775167      392193    5  Extended
/dev/xvde5        15990784    16775167      392192   82  Linux swap

And you can do whatever you need inside. Note that network is also working.

[^n]: Same distro, e.g any Debian on top of a previous Debian