Xen Orchestra 5.82

We're super excited to bring you the latest and greatest Xen Orchestra release - version 5.82. Trust us, we were just as surprised as you when we realized this "little" update turned out to be jam-packed with awesome new features. So buckle up, and let's dive in to explore the fabulous world of Xen Orchestra 5.82!

💾 Backup

Get ready, because this release is all about leveling up your backup game! We've made significant improvements and introduced some essential naming changes that'll pave the way for even more awesome updates in the future. Let's take a closer look at what's new in the world of Xen Orchestra backups.

Advanced backup restore check

Remember when we introduced VM restore health checks a year ago? It was a game-changer, and we took it a step further with the "Auto restore check" just a month later. Well, fasten your seatbelts, because we're going even deeper now with the "Advanced backup restore check" (a.k.a. "Application aware restore check").

Let's break it down. During the usual backup restore check, the "restored" VM is booted on the SR of your choice. In the past, we'd only wait for the guest tools to be available, which indicated that the OS had booted successfully. Now, we're also waiting for a custom script to run inside your VM, allowing you to test your local app or service and report whether everything is A-OK (or not). Magic happens via the XenStore, meaning Xen Orchestra can be aware of the result of your check.

To set this up, you'll need to do the following:

1. Add a tag xo-backup-health-check-xenstore on the VM.
2. Make sure the VM is part of a job with health check enabled.
3. Ensure the script starts automatically on VM boot.

Keep in mind, there won't be any network access during the check. We perform it offline to avoid interfering with your production VM.

Example: testing a SQLite Database

To give you an idea of how to create your own checks, let's walk through an example script. In this Bash script, we'll test whether a SQLite database is present and working properly:

#!/bin/sh

# must be executed as root to be able to use xenstore-read and xenstore-write

# fail in case of error or undefined variable
set -eu

# stop there if a health check is not in progress
if [ "$(xenstore-read vm-data/xo-backup-health-check 2>&1)" != planned ]
then
  exit
fi

# not necessary, but informs XO that this script has started which helps diagnose issues
xenstore-write vm-data/xo-backup-health-check running

# put your test here
#
# in this example, the command `sqlite3` is used to validate the health of a database
# and its output is captured and passed to XO via the XenStore in case of error
if output=$(sqlite3 ~/my-database.sqlite3 .table 2>&1)
then
  # inform XO everything is ok
  xenstore-write vm-data/xo-backup-health-check success
else
  # inform XO there is an issue
  xenstore-write vm-data/xo-backup-health-check failure

  # more info about the issue can be written to `vm-data/health-check-error`
  #
  # it will be shown in XO
  xenstore-write vm-data/xo-backup-health-check-error "$output"
fi

It's pretty straightforward, but let's do a quick analysis of this example:

1. if [ "$(xenstore-read vm-data/xo-backup-health-check 2>&1)" != planned ] means this check won't be done on a regular boot, ONLY when used in the backup restore health check process. That's why you can confidently have this script on boot.
2. if output=$(sqlite3 ~/my-database.sqlite3 .table 2>&1) means we'll list the tables in a SQLite database. If the DB is fine, the command will return 0 (success) then we'll write that in the XenStore with xenstore-write vm-data/xo-backup-health-check success
3. And if it's a failure, then: xenstore-write vm-data/xo-backup-health-check failure. We can even provide the detailed error via xenstore-write vm-data/xo-backup-health-check-error "$output" so your backup job will report it!

Definitely, any other language can do the same thing, as long as you are able to write into the XenStore using the right keys.

Introducing block retry for NBD backup

We've got another great addition for you in this release: the implementation of block retry for NBD backups. If you're using NBD-enabled backups, you already know they're fast and light on Dom0 resources. But now, they're even more robust!

The beauty of NBD is that, unlike the VHD handler for exporting VM disks, it allows us to retry fetching blocks without losing the entire transfer. Currently, our retry capability is hard-coded to retry 5 times, but we'll make it configurable in the future.

So why is block retry such a big deal? With thousands of XO users worldwide performing backups every day, there's always a chance that something could go wrong with the infrastructure – a faulty switch, cable, network interface, internet link, or even a memory issue on the Dom0. With block retry, backups can now handle transfer or network issues, and keep everything running smoothly in a fully transparent fashion!

Get ready to experience an even more reliable and resilient backup process.

Changes to come in backup naming

We're always looking to improve and refine Xen Orchestra, and that includes making things clearer for our users. Over time, we've added features one after another, sometimes without fully anticipating what might come next. This has led to some confusion in our backup naming conventions, so we're making a few changes to set things straight.

Here's a short recap:

Previously, the term "full" could refer to either a full backup (XVA format) or the initial run of our former "delta" backup. And "delta" was another run of our Delta Backup – talk about confusing! 🙃

To clear things up, we're introducing the following naming changes:

• The first run of an Incremental Backup will be called the base.
• Subsequent runs of an Incremental Backup will be called delta (as before).
• A full backup will only refer to a backup from the full backup feature.

For replication, we'll use similar naming conventions:

• A Disaster Recovery job will be called full replication.
• The first run of an incremental replication will be called the base.
• Subsequent runs of an incremental replication will be called delta.

These changes should make it easier to understand and discuss Xen Orchestra backups and replications, setting the stage for even more exciting features in future releases. Enjoy the newfound clarity!

❄️ Smart host reboot

Or also known as "Reboot your host without losing VM memory".

For those using a pool with shared storage, rebooting a host is a breeze; VMs are migrated to other nodes seamlessly. This is also true for our Rolling Pool Upgrade mechanism. But what if you don't have shared storage or only have a single host? How can you reboot without losing VM memory or having to reboot the VMs themselves?

We've got you covered with our new Smart Host Reboot feature! The solution is to freeze your VMs (using suspend to RAM) during the host reboot and resume them once the host is back. With our new dedicated button, you no longer have to go through the cumbersome manual process:

But wait, there's more! If you only have one host, your Xen Orchestra VM is likely running locally. If we suspend it, the Smart Host Reboot process would be interrupted. Fear not! We've managed to detect if a VM is Xen Orchestra and exclude it from the process. While the XOA VM will reboot, its autopower_on setting allows it to start back up, detect the previously suspended VMs, and resume them.

Now, you can reboot a single host without any shared storage (e.g., for security updates) without causing a VM reboot from the VM OS perspective. There's no excuse to avoid applying updates anymore!

☸️ Project Pyrgos (k8s)

If you missed the original announcement about what is Project Pyrgos, you can catch up here:

Announcing project Pyrgos

Have you ever wanted to easily create and update Kubernetes clusters on top of your on-prem infrastructure? That’s exactly what Pyrgos project is.

Xen Orchestra BlogOlivier Lambert

Ability to create multiple control plane

You can now create multiple control plane, to remove a previous SPOF. It's a good way to play with Kubernetes and kill some control plane VMs and observe the result. In any case, now you can make your cluster resilient from one click.

Naming modification

We also updated some wording to match the current Kubernetes official naming convention, like "Control plane" instead of "Master Node". It seems rather a small change, but it's important to keep up on how things are called!

🔭 XO Lite

We're not just focused on rolling out a ton of Xen Orchestra features every month – we're also committed to improving XO Lite! While the latest enhancements are more behind-the-scenes, they're still significant and contribute to an even better user experience.

Object subscriptions

Previously, when you loaded XO Lite, you had to wait for all XAPI objects to load before the page was displayed, much like how regular Xen Orchestra works. This approach was inefficient, slow, and resource-consuming. That's where object subscriptions come in to save the day!

With object subscriptions, each displayed component individually subscribes to the data it requires. For example, if a table component lists VMs, it will "ask" to load only the VM objects and nothing else. If another component on the same page requires the same objects, they will share the collection.

When you navigate to another page (e.g., Host) that doesn't need the VM collection, it will no longer be used. This new approach is highly efficient and fast, offering a much-improved user experience in XO Lite.

New components incoming

We've got even more in store for you! Next month, we'll be unveiling a host of new components and features for XO Lite, adding even more value to your experience. Keep an eye out for these upcoming enhancements, and stay tuned for more updates as we continue to improve and expand XO Lite's capabilities. Get ready for some exciting times ahead!

📡 REST API

We're excited to announce that our REST API is now more stable and has officially exited its alpha status! We're thrilled with its performance and simplicity, and we're confident you'll appreciate it too.

Technical improvements

Our REST API is now capable of streaming objects, which means less memory usage for large collections. Additionally, we start streaming objects as soon as possible, rather than waiting for all objects to load. This enhancement provides a more efficient and faster experience.

Quick tips

As the REST API continues to gain traction, we'd like to share some tips and tricks to help you unlock its full potential.

Browsing the REST API

If you've already accessed the XO web UI previously, you already have a token. This means you can easily browse the REST API directly from your current browser and explore the available data. This convenient access allows you to better understand the API's capabilities and how to use it effectively.

An example script to list VMs without any backup

What about creating a simple script to list all the VMs that are not in any backup job? Here is an example in Javascript (but any programming language will do it):

https://gist.github.com/olivierlambert/a7d4a0add61c96d837db81970bc7212f

This will return a CSV output that you can use for any inventory work you have to do, example:

# node restapi.mjs <XOA URL> <XOA TOKEN>
"VM-NAME";"VM-UUID"; "HOST/CLUSTER";"Tags"
"Test Debian 11";"29d19fd6-6337-2f5c-1dd8-789e2463271d";"c1d386df-990a-a892-ca89-267e7b3b5afd/c1d386df-990a-a892-ca89-267e7b3b5afd";"Prod"
"dml-netperf-zen4-1";"b85e7d22-c63d-3fcf-3f3c-5f22e599d2c8";"6362271f-abdc-40ac-a59b-a447c2570ed1/c1d386df-990a-a892-ca89-267e7b3b5afd";"Test"

It's just a simple example how requesting the REST API can be used. Obviously, there's many many other ways to use it! We created a dedicated thread in our forum so we can all share our scripts:

XO REST API: share your tips and scripts!

As our REST API is going better and better, we wanted to centralize the community efforts on building scripts and solutions on top of it Our first example on our side, is to help returning a CSV with all VMs without any backup job: https://gist.github.co…

XCP-ngolivierlambert

🆕 Misc

In addition to the main announcements, we're also introducing several quality of life features and enhancements.

Eject the CD on template conversion

When converting a VM, it's possible you might forget to remove the inserted ISO, which can cause issues when using the template later. To avoid this problem, the ISO is now automatically ejected before the template conversion. Thank you, Cécile, for the idea/report! 👍

Identify VM creator and template

As an admin, you can now easily see which user created a VM and from which template, directly in the VM overview.

Compatibility with older XenServer

Eight years ago, Citrix announced XenServer 6.5, which brought many improvements, including a new XAPI transport capability: JSON-RPC instead of XML-RPC. This change significantly improved Xen Orchestra's performance, thanks to JSON being faster to parse than XML. Citrix also added a mixed mode called "JSON in XML-RPC."

Improving Xen Orchestra performances

Xen Orchestra is now going 250 times faster. Period.

Xen Orchestra BlogOlivier Lambert

In Xen Orchestra, we built an "auto" transport capability that adapts to the XAPI's response. We always started with JSON-RPC (the best protocol) and downgraded to JSON inside XML-RPC or XML-RPC if there were errors. However, we decided to create a more precise fallback mode this year.

Even though JSON-RPC works for most operations, there are some subtle bugs in XAPI's transport implementation. These issues are mainly related to improper integer and float number casting. We've seen these bugs up to XenServer 7.1 (which is EoL since). Since we know some users still run older XenServer versions, we added a specific option to force the transport to XML-RPC for any XenServer versions earlier than 7.1:

[xapiOptions]
transport = 'xml-rpc'

XO config compression

When you backup or export your XO configuration, it's now compressed by default. It's a rather minor feature in terms of size impact, but in ratio it's 50% saved at least. Not bad! This also open the door to a future exciting feature. But shhh! 🤫 You will discover it next month.

Display free space on your SRs

In the Dashboard/Health view, we added a row showing the free space available on your Storage Repositories: