Meltdown and Spectre for XenServer

A recap on how Spectre and Meltdown can affect your Citrix XenServer infrastructure. And a full guide on how to mitigate it.

If you weren't in a desert this week, you probably heard about Meltdown and Spectre vulnerabilities. But how it can affect your infrastructure running Citrix XenServer?

Meltdown

What's the problem?

Someone having access to the VM will be able to read the memory outside the VM itself, and therefore read data from other VMs on this same physical host (let's call them "untrusted VMs"). You are NOT affected if you only have trusted VMs, but note that an attacker could enter via other application exploits (web server etc.) on your VMs. In short, you just lost a protection in a defense-in-depth. Not good.

Meltdown is the easiest exploit to run (we'll see that Spectre is more subtle to exploit but also to "solve").

Am I affected?

Meltdown is using a design flaw into Intel CPUs only. This is called by Xen sec team "SP3" (aka rogue data cache load). You are impacted only if you are using:

• 64-bits PV type VM (HVM/PVHVM aren't affected!)
• Intel CPUs (AMD chip design is a bit different and not affected)
• untrusted VMs, ie untrusted users having VM access (even non-root!)
• All XenServer versions are affected

64-bits PV guests are vulnerable because guest and hypervisor share the same address space, but with different privileges. HVM aren't.

What can I do?

You can mitigate the risk by migrating your PV guest to HVM. To know if a VM is PV or HVM, you can check it in the VM view of Xen Orchestra, eg here is one PV and one HVM guest:

Or even better, use the filter virtualizationMode:pv to see them all directly:

If you don't have Xen Orchestra yet, you can use xe command to list all your PV guests with xe vm-list HVM-boot-policy= is-control-domain=false:

uuid ( RO)           : 49e3ee6c-f387-fb91-3b2c-1aa52f20c6a7
     name-label ( RW): XenServer DDK 7.1.0-137272c import
    power-state ( RO): running


uuid ( RO)           : 2c49f8a1-1139-6dd3-7869-5ddc5ddd6be1
     name-label ( RW): XenServer DDK 7.0.0-125770c import
    power-state ( RO): running


uuid ( RO)           : 7dba3f40-eeef-5398-e21a-756418368830
     name-label ( RW): Citrix License Server Virtual Appliance v11.14.0.1
    power-state ( RO): running

Then you can migrate your Citrix XenServer VMs from PV to HVM!

Migrating from PV to HVM

You need to have, in your VM:

• Compatible kernel (2.6.24)
• Grub installed in the MBR (eg grub-install)

Then you just have to change the HVM-boot-policty value of the VM:

xe vm-param-set HVM-boot-policy="BIOS\ order" uuid=<VM_UUID>

Halt and start your VM, you are now in HVM. If something is wrong, empty the previous parameter, shutdown and start again to be back in PV.

Inside the VM

Solving/mitigating the hypervisor won't solve your issue on the VM level: remember to upgrade your VM OS too, or untrusted users could read part of the VM memory that you don't want to!

Spectre

What's the problem?

Like Meltdown, someone having access to the VM will be able to read the memory outside the VM itself, and therefore read data from other VMs on this same physical host (let's call them "untrusted VMs"). Again, you are NOT affected if you only have trusted VMs, but note that an attacker could enter via other application exploits (web server etc.) on your VMs.

Spectre is the NOT easy to exploit (more complicated and slow to access memory, something like 1500 bytes/second). But it's also trickier to mitigate it and affects everyone.

Am I affected?

Short answer: yes. Spectre contains both "SP2", aka "Branch Target Injection" and "SP1" aka "Bounds-check bypass".

For now, there is no known exploit for "SP1". The issue then is on "SP2". If you have untrusted VMs, you are vulnerable, whatever XenServer version you are using, or the virtualization mode.

What can I do?

On XenServer 7.x

All XenServer 7.x versions have now a patch from Citrix. In Xen Orchestra, you should see it in Dashboard view, or pool view. It's also available on any latest XS version (7.3).

Note: XenServer 7.1 patch (same for 7.0), needs a Citrix license. Another reason to take a look on XCP-ng to avoid a paywall for critical security issues, right?

Update: A "free" patch was released for 7.2, due to the massive security issue (you can install it on XS free version).

On XenServer older than 7.x

You will need to upgrade! If you are in an isolated environment, the risk will be lower but keep in mind you have a potential breach that won't be fixed.

Conclusion

All those vulnerabilities affect infrastructure with untrusted VMs. If you run your on-premise/isolated "cloud", you are relatively safe (still losing one "defense" anyway). This must be a reminder that even on the cloud, your applications still run on some hardware.

Remember that having VMs on a public cloud is like having multiple unknown random flatmate.