Release

Xen Orchestra 5.89

Xen Orchestra 5.89 makes a splash with its latest release, featuring over 20 dynamic new features, blending innovation and user-centric enhancements in a powerful update to round off the year.

Olivier Lambert

Nov 30, 2023 • 12 min read

It's time for a new release! November is now almost done, last stop before December release and the end of 2023. Despite it's not Christmas yet, we managed to deliver more than 20 ~~presents~~ new features in one month!

🧑‍🚀 Project & Community

In just a single month, many things happened. The most notable is Broadcom's monumental acquisition of VMware for over $60 billion, a move that has resonated across the tech industry. However, the developments don't stop there: Xen Orchestra team is expanding and Xen 4.18 landed alongside numerous other news.

The new virtualization landscape

In this blogpost I wrote (Olivier' speaking), I dug into the staggering $100 billion worth of acquisitions and consolidations in the virtualization market, a phenomenon that's causing a seismic shift in the industry:

Xen 4.18 is here

The latest Xen Project hypervisor version just landed, in 4.18. We are proud to demonstrate our commitment upstream since one of our developer made into the Top 10 contributors!

XO team still growing

Bastien has recently become a part of our team, primarily focusing on the backend part of XO. His addition is a significant boost to our efforts, as he collaborates closely with Julien and Florent to accelerate our development pace. We also now have a dedicated person to get focus on Project Pyrgos (k8s cluster creation/lifecycle directly from Xen Orchestra), so expect new features for the next releases!

Community feedback on our Rust Linux tools

If you missed the announcement, take a look here:

And here:

There's now a dedicated thread where you can test our new tools: https://xcp-ng.org/forum/topic/7974/new-rust-xen-guest-tools/

Please test and provide your feedback!

XO Lite changelog

We have now a dedicated changelog for XO Lite! Feel free to take a look: https://github.com/vatesfr/xen-orchestra/blob/master/%40xen-orchestra/lite/CHANGELOG.md#changelog

💾 Backup

Backup functionality has always been a top priority for us in Xen Orchestra. This month, we're excited to announce the release of several new features and various enhancements!

Differential restore

Our XO Backups features have always stand out as a very flexible tool, allowing you to restore a VM to any location on any storage, without any prerequisites. However, we've identified and enhanced a specific scenario: restoring your VM to a storage unit that already houses your original VM. Our latest improvement significantly cuts down the restoration time for this use case.

Here's how it works: instead of a full restore, we utilize the existing VM disk or snapshot as a base, restoring only the differential data to a new disk. This approach drastically reduces restore time, particularly for large VMs. For instance, with a transfer rate of 60MiB/s, restoring a 200GiB VM would typically take about an hour. But with our differential restore method, even a 600GiB disk can be restored in a fraction of the time: mere minutes instead of hours.

Most importantly, this process is designed with data integrity in mind. The original VM disk remains untouched during the restore; we only read from the latest snapshot to use it as a foundation for creating the new VM and disk.

Show backup date and description of restored VM

Restoring a VM often requires extensive context, especially when navigating through multiple versions to find the correct one.

To simplify this process, we managed to add more information in the restored VM. This includes the original description of the VM, supplemented with useful data like the time of restoration and the source Backup Repository (BR).

This feature is particularly useful in complex environments where multiple restorations are common, ensuring that each restored VM is easily identifiable and its lineage is clearly documented.

This addition ensures that you can confidently identify and utilize the specific version of the VM you intended to restore.

API/CLI file level restore

Xen Orchestra already offers a convenient feature for file-level restoration directly through the XO user interface. This is particularly handy when you need to retrieve specific files without restoring the entire VM. Simply select the files you need, download them, and you'll receive a ZIP file right in your browser!

However, there are scenarios where you might want to transfer these files to a different destination or use tools like rsync to synchronize them with files on an existing VM.

Let's explore how to do so with xo-cli! First, we need to list the remotes:

$ xo-cli remote.getAll
[
  {
    benchmarks: [
      {
        readRate: 370947830.1903045,
        timestamp: 1644844831515,
        writeRate: 107656040.17545153
      }
    ],
    enabled: true,
    id: '6d77e904-b28d-4055-b1cb-041df77c9e2c',
    name: 'Local',
    url: 'file:///srv/xo-backups'
  }
]

Now we'll list all the backups on this remote:

$ xo-cli backupNg.listVmBackups remotes=json:'["6d77e904-b28d-4055-b1cb-041df77c9e2c"]'
{
  '6d77e904-b28d-4055-b1cb-041df77c9e2c': {
    '123e4f2b-498e-d0af-15ae-f835a1e9f59f': [
      {
        disks: [
          {
            id: '/xo-vm-backups/123e4f2b-498e-d0af-15ae-f835a1e9f59f/vdis/86898a59-15bd-463c-91e9-ebc55f42adf5/2da1102c-9be3-43a0-82f5-9111e362c66f/20231116T120859Z.vhd',
            name: 'backup QA - system',
            uuid: '80dfc4ec-41f3-4d0f-823c-61d4287ca628'
          },
          {
            id: '/xo-vm-backups/123e4f2b-498e-d0af-15ae-f835a1e9f59f/vdis/86898a59-15bd-463c-91e9-ebc55f42adf5/0a1d2e50-5a59-47e4-b0bd-4bbff7765d86/20231116T120859Z.vhd',
            name: 'backup QA - data',
            uuid: 'a449d2c3-c457-41bf-bb2b-82e9cb09aaab'
          }
        ],
        id: '6d77e904-b28d-4055-b1cb-041df77c9e2c//xo-vm-backups/123e4f2b-498e-d0af-15ae-f835a1e9f59f/20231116T120859Z.json',
        jobId: '86898a59-15bd-463c-91e9-ebc55f42adf5',
        mode: 'delta',
        scheduleId: '58f593e1-afa4-43f9-85f7-5bf8ed199584',
        size: 1533407744,
        timestamp: 1700136539071,
        vm: {
          name_description: 'Cloud Ready from XO',
          name_label: 'backup QA'
        }
      }
    ]
}

Great! Now we want to specifically get the list of partitions for a disk:

$ xo-cli backupNg.listPartitions remote=6d77e904-b28d-4055-b1cb-041df77c9e2c disk=/xo-vm-backups/123e4f2b-498e-d0af-15ae-f835a1e9f59f/vdis/86898a59-15bd-463c-91e9-ebc55f42adf5/2da1102c-9be3-43a0-82f5-9111e362c66f/20231116T120859Z.vhd
[
  {
    id: '8d8f3f0f-01',
    name: '',
    nr: '1',
    size: 4292870144,
    start: 2048,
    type: 131
  }
]

Almost there! Now, we'll ask to mount the partition in XOA so we can access the files:

$ xo-cli backupNg.mountPartition remote=6d77e904-b28d-4055-b1cb-041df77c9e2c disk=/xo-vm-backups/123e4f2b-498e-d0af-15ae-f835a1e9f59f/vdis/86898a59-15bd-463c-91e9-ebc55f42adf5/2da1102c-9be3-43a0-82f5-9111e362c66f/20231116T120859Z.vhd partition=8d8f3f0f-01
/tmp/bujxvyk72mr

It's now mounted! If you want to check the currently mounted partition, a simple call will do the trick:

$ xo-cli backupNg.listMountedPartitions
[
  {
    disk: '/xo-vm-backups/123e4f2b-498e-d0af-15ae-f835a1e9f59f/vdis/86898a59-15bd-463c-91e9-ebc55f42adf5/2da1102c-9be3-43a0-82f5-9111e362c66f/20231116T120859Z.vhd',
    partition: '8d8f3f0f-01',
    path: '/tmp/bujxvyk72mr',
    remote: '6d77e904-b28d-4055-b1cb-041df77c9e2c'
  }
]

When it's done, just unmount it:

$ xo-cli backupNg.unmountPartition remote=6d77e904-b28d-4055-b1cb-041df77c9e2c disk=/xo-vm-backups/123e4f2b-498e-d0af-15ae-f835a1e9f59f/vdis/86898a59-15bd-463c-91e9-ebc55f42adf5/2da1102c-9be3-43a0-82f5-9111e362c66f/20231116T120859Z.vhd partition=8d8f3f0f-01
true

The unmount time in Xen Orchestra is set with a 5-minute timer, which activates after a period of inactivity. This feature is designed to ensure that the unmounting process does not interrupt any ongoing file restoration activities. If, for any reason, you forget to manually unmount, Xen Orchestra takes care of it by automatically unmounting after 24 hours.

🔭 XO Lite

This month has been significant for XO Lite, with various enhancements made to its functionality. We're proud to witness the evolution of Xen Orchestra's "little brother," as it becomes increasingly capable and user-friendly for a variety of tasks.

A more compact UI

Following valuable input from our community, we decided to optimize space usage in Xen Orchestra by crafting a more compact overall interface. The result is a fine balance between compactness and usability. To appreciate the improvements, take a look at the "before and after" comparison!

The recent interface optimizations in Xen Orchestra have led to a notable increase in efficiency, particularly evident in the tree view of VMs. Previously, the interface could display approximately 18 VMs, but with our latest updates, it can now show up to 24 VMs! (+30%)

We've also refined the dashboard side of the interface. It now provides more information at a glance, enhancing the user's ability to monitor and manage their resources efficiently.

XVA export

You can now export a VM directly from XO Lite, as an XVA file you can download in your browser:

But it's also possible to download multiple VMs at once! For example:

It even comes with the capability to detect if your browser is blocking pop-ups:

VM migration

You can now live migrate a VM inside a pool, directly from XO Lite UI:

Connecting on a pool member

To enhance navigation and clarity when accessing pool members. If you open the URL of a pool member that is NOT the pool master, this status is now clearly displayed on the interface.

Moreover, for added convenience, there is a direct link provided to navigate to the pool master.

Redirecting XO Lite to another master

This one is fun: you can use the XO Lite for one pool to then decide to connect to another pool that does NOT have XO Lite installed!

📡 REST API

Many updates on the REST API side of things. These improvements are geared towards making the API even more versatile and effective for a wide range of operations and automation tasks.

Display and list XO users

You can now list all your XO users with a simple:

curl \
  -X GET \
  -b authenticationToken=KQxQdm2vMiv7j \
  'https://xo.company.lan/rest/v0/users' \

This will return an array of users:

[
"/rest/v0/users/fbd24be9-3ed6-498e-a0a0-62badf7564ff",
"/rest/v0/users/421c8487-3aa5-4281-b5d9-dadafb9c0914"
]

You can then display details on one user, even get his public SSH keys!

{
  "id": "421c8487-3aa5-4281-b5d9-dadafb9c0914",
  "email": "user@example.org",
  "groups": [],
  "permission": "admin",
  "preferences": {
    "sshKeys": [
      {
        "title": "mykey@mypc",
        "key": "ssh-rsa AAAAB3NzaC1yc2EAA[...] mykey@mypc"
      }
    ],
    "filters": {
      "VM": {
        "test": "power_state:running"
      }
  }

Import a VDI content

You can now import a VHD or raw file content into an existing VDI. The only limitation is the size of the VDI must match the size of the VDI previously exported:

curl \
  -X PUT \
  -b authenticationToken=KQxQdm2vMiv7j \
  -T myDisk.vhd \
  'https://xo.company.lan/rest/v0/vdis/1a269782-ea93-4c4c-897a-475365f7b674.vhd' \
  | cat

This might be pretty useful to inject whatever content you have, for example in some automation scripts to get back on a disk state you had locally.

Import an XVA

You can now import an XVA file directly into XO via the REST API. Here is an example with curl:

curl \
  -X POST \
  -b authenticationToken=KQxQdm2vMiv \
  -T myVm.xva \
  'https://xo.company.lan/rest/v0/pools/355ee47d-ff4c-4924-3db2-fd86ae629676/vms \
  | cat

This will import the XVA into the pool default storage repository (SR). If you want to specificy the target SR, you can simple add the SR UUID in the URL: ?sr=86a9757d-9c05-9fe0-e79a-8243cb1f37f3

💡

As usual, the final | cat ensures cURL's standard output is not a TTY, which is necessary for upload stats to be dislayed.

Tags management

You can now edit or remove a tag very easily. It's using what we call the "collection", and it works that way:

To add a tag:

curl \
-X PUT \
-b authenticationToken=KQxQdm2vMiv7jB \
'https://xo.company.lan/rest/v0/vms/770aa52a-fd42-8faf-f167-8c5c4a237cac/tags/My%20tag'

Removing a tag is also trivial:

curl \
  -X DELETE \
  -b authenticationToken=KQxQdm2vMiv7jB \
  'https://xo.company.lan/rest/v0/vms/770aa52a-fd42-8faf-f167-8c5c4a237cac/tags/My%20tag'

Relying on tags is a great way to identify resources and apply actions on them automatically with whatever logic you need in your system.

🗃️ Netbox

Last July, we already supercharged our Netbox plugin, see this as a reminder:

We're taking our integration with Netbox to the next level by introducing an optional synchronization feature between your XO users and the "tenant" entity in Netbox. This feature allows for the tracking of user activities within Netbox, specifically enabling you to see which user created which VM!

Don't forget to add the uuid field for the Tenancy>Tenant object in Netbox configuration if you want to do that sync:

🚀 XCP-ng 8.3 features

In this section, we're focusing on the new features that have been added to Xen Orchestra, specifically tailored for compatibility with XCP-ng 8.3. It's important to note that most of the recent modifications made in XCP-ng are not backported to the Long-Term Support (LTS) version, for obvious stability reasons.

Detailed task for coalesce

A key enhancement in XCP-ng 8.3 is the ability to closely monitor storage coalescing tasks. What sets this feature apart is the added functionality where each coalescing task is directly linked to the specific pool and Storage Repository (SR) involved in the process. This direct linkage makes it much easier to understand the ongoing coalescing activities and allows for quick exploration and management of the associated pool and SR.

🆕 Misc

The "Misc" section this month is still quite substantial, despite the numerous other features we've already delivered. And we will continue to raise the pace!

Ignore empty VDIs

One of the useful aspects of the dashboard/health view in Xen Orchestra is its ability to highlight potential misconfigurations or issues with your disks. However, VDIs with a size of 0, such as udev devices or other special devices, should not clutter this view. We've now updated the system to exclude these empty VDIs from the dashboard/health view, ensuring that only relevant data is displayed.

Better token management

Each time you login, you have a token. It's manageable, but results in a long list of tokens displayed to the user. We've already fixed this on the backend, and now we've made improvements on the UI front as well.

Now, we're reusing tokens throughout their duration, and we're providing more detailed information about each tokens:

The user agent through which you logged in
The time of login

This means there's now one token per "device" from which you access XO, typically a web browser. This enhancement greatly improves both security and transparency, allowing you to clearly see the details of each login and the device used, streamlining token management and enhancing overall system security.

Boot a VM with disk with an ISO

You can now boot a VM with an existing disk using an ISO file. Previously, this wasn't considered necessary since a disk with data, like a template with an operating system, was thought to be sufficient. However, based on community feedback highlighting certain use cases, this feature has been added for increased flexibility.

Disabling the console view

In XenCenter, if you add the key other-config:disable_pv_vnc=1 to the VM object, the console view for the user is hidden. To increase compatibility and interoperability between XenCenter and Xen Orchestra, we have also implemented this feature. When this configuration is set, the console will be hidden in Xen Orchestra as well.

Show disk import status in labels

When importing a disk or a VM with disks, the name of the drive is now altered during the import phase. This update removes any confusion about whether a disk is currently being imported or not.

Show import task progress in the disk views

In earlier versions, tracking the progress of an imported VM (either directly or via backup restore) required going to the global task view. Now, you can see the progress of importing disks directly in the Disk tab of the VM being imported, making it easier to monitor the import process.

Storage forget: clearer message

When deciding to forget a SR, a clearer warning message now informs you about the consequences of this action. Additionally, to confirm the action, you need to type "Forget this SR," which helps prevent accidental validations.

The complete changelog for XO is available here.