XO documentation

# Advanced features

This section is dedicated to all others Xen Orchestra "advanced features".

# Live Telemetry

When you are working on your virtualized infrastructure, you might need a very detailed view on what's going on. From resource consumption analysis (CPU/RAM) to network traffic, but also storage, having a kind of very detailed dashboard per host can be really useful.

TIP

This feature will be only available in XCP-ng+XOA. If you are a hardcore Citrix Hypervisor user, let us know if you want this.

# Enable Advanced Live Telemetry for a host

Go into the Advanced host view:

By clicking here, XOA will handle everything: installing netdata on the host, and configure a secured/encrypted streaming toward your XOA.

As soon it's installed, the button will now be replaced by a "Go to Advanced Live Telemetry". As soon you'll click on it, you'll be redirected to the netdata page of the host. If you have multiple host, you can access any of them directly from the top menu of the netdata UI.

# Examples

IMPI info (power, temperatures, voltages, fans…):

TIP

This feature is only available in one-click in your XOA. However, all sources and packages to do it yourself are available!

Memory details:

Firewall:

# Cloud init

Cloud-init is a program "that handles the early initialization of a cloud instance"[^n]. In other words, you can, on a "cloud-init"-ready template VM, pass a lot of data at first boot:

  • setting the hostname
  • add ssh keys
  • automatically grow the file system
  • create users
  • and a lot more!

This tool is pretty standard and used everywhere. A lot of existing cloud templates are using it.

So it means very easily customizing your VM when you create it from a compatible template. It brings you closer to the "instance" principle, like in Amazon cloud or OpenStack.

# Requirements

You only need to use a template of a VM with CloudInit installed inside it. Check this blog post to learn how to install CloudInit (opens new window).

TIP

In XOA 5.31, we changed the cloud-init config drive type from OpenStack (opens new window) to the NoCloud (opens new window) type. This will allow us to pass network configuration to VMs in the future. For 99% of users, including default cloud-init installs, this change will have no effect. However if you have previously modified your cloud-init installation in a VM template to only look for openstack drive types (for instance with the datasource_list setting in /etc/cloud/cloud.cfg) you need to modify it to also look for nocloud.

# Usage

First, select your compatible template (CloudInit ready) and name it:

Then, activate the config drive and insert your SSH key. Or you can also use a custom CloudInit configuration:

CloudInit configuration examples are available here (opens new window).

You can extend the disk size (in this case, the template disk was 8 GiB originally). We'll extend it to 20GiB:

Finally, create the VM:

Now start the VM and SSH to its IP:

  • the system has the right VM hostname (from VM name)
  • you don't need to use a password to access it (thanks to your SSH key):
$ ssh centos@192.168.100.226
[centos@tmp-app1 ~]$

The default cloud-init configuration can allow you to be to be a sudoer directly:

[centos@tmp-app1 ~]$ sudo -s
[root@tmp-app1 centos]#

Check the root file system size: indeed, it was automatically increased to what you need:

[centos@tmp-app1 ~]$ df -h
/dev/xvda1          20G    1,2G   18G   6% /

# Web hooks

WARNING

We need your feedback on this feature!

# Configuration

The plugin "web-hooks" needs to be installed and loaded for this feature to work.

You can trigger an HTTP POST request to a URL when a Xen Orchestra API method is called or when a backup job runs.

  • Go to Settings > Plugins > Web hooks
  • Add new hooks
  • For each hook, configure:
    • Method: the XO API method that will trigger the HTTP request when called. For backup jobs, choose backupNg.runJob.
    • Type:
      • pre: the request will be sent when the method is called
      • post: the request will be sent after the method action is completed
      • pre/post: both
    • URL: the full URL which the requests will be sent to
    • Wait for response: you can choose to wait for the web hook response before the method is actually called ("pre" hooks only). This can be useful if you need to automatically run some tasks before a certain method is called.
  • Save the plugin configuration

From now on, a request will be sent to the corresponding URLs when a configured method is called by an XO client.

# Request content

POST / HTTP/1.1
Content-Type: application/json

The request's body is a JSON string representing an object with the following properties:

  • type: "pre" or "post"
  • callId: unique ID for this call to help match a pre-call and a post-call
  • userId: unique internal ID of the user who performed the call
  • userName: login/e-mail address of the user who performed the call
  • method: name of the method that was called (e.g. "vm.start")
  • params: call parameters (object)
  • timestamp: epoch timestamp of the beginning ("pre") or end ("post") of the call in ms
  • duration: duration of the call in ms ("post" hooks only)
  • result: call result on success ("post" hooks only)
  • error: call result on error ("post" hooks only)

# Request handling

Quick Node.js example of how you may want to handle the requests

const http = require('http')
const { exec } = require('child_process')

http
  .createServer((req, res) => {
    let body = ''
    req.on('data', chunk => {
      body += chunk
    })
    req.on('end', () => handleHook(body))
    res.end()
  })
  .listen(3000)

const handleHook = data => {
  const { method, params, type, result, error, timestamp } = JSON.parse(data)

  // Log it
  console.log(`${new Date(timestamp).toISOString()} [${method}|${type}] ${params} → ${result || error}`)

  // Run scripts
  exec(`./hook-scripts/${method}-${type}.sh`)
}

# Alerts

Alerts are a way to warn the administrator about various events. The first kind of alerts will be emails and also in a dedicated area of xo-web to display them.

# Performances alerts

The administrator will configure alerts based on performance thresholds.

The configurable metrics are:

  • CPU usage (VM, host)
  • RAM usage (VM, host)
  • network bandwidth (VM, host)
  • load average (host)
  • disk IO (VM)
  • total IO (SR, only for XenServer Dundee and higher)

If any configured values exceed the threshold during a selected period of time, an alert will be sent.

Those alerts will be also stored and accessible in the web interface, and also later for the load balancing feature (helping it to solve those performance problems).

# Updates alerts

WARNING

👷 This feature might be missing. We are investigating. 👷

When your XOA detects new packages, you'll be notified by email.

# Backup alerts

Same story for backups: if a backup fails, you'll receive an email.

You can choose to be notified only if it fails or even after each backup job.

Current supported alerts system:

  • Email
  • XMPP

# Job manager

The key idea is to be able to schedule any action (migrate, reboot etc.), for things like backups, snapshots or DR actions.

In the main menu, go to the "Job Manager" section:

You can now schedule all actions on your hosts, VMs, or ACLs. It's configured in 2 steps:

  1. Create a job
  2. Schedule it!

Real example, step by step: Creating a job called "security reboot" (in this case, restarting "nfs" and "Core1" VMs):

Note that you can execute this job now by clicking on the orange play button (to test it for instance):

Schedule the job (every Sunday at 5:00 AM):

And that's it! The job is listed in the Overview:

The possibilities are infinite! You can schedule a lot of things (any actions on a VM, like migrate, start, clone, suspend etc. Same thing also applies to hosts).

# Examples

# Save on your electric bill

  • plan a live migration of your VMs at 11:00PM to a less powerful host, then shutdown the big one
  • start the big server at 6:00AM and migrate the VMs back 15 minutes later

# Scale when needed

  • schedule the boot of extra VMs during your usual activity spikes (horizontal scaling)
  • also add more vCPUs or RAM to these VMs at the same time
  • go back to the previous state when your planned load is low (e.g: during the night)

# Planned reboot

  • For example: your client app is not very stable, or you need to reboot every month after kernel updates: schedule this during the weekend!

# Add or Remove ACLs

  • revoke your user ACLs Friday at 11:00PM (e.g: no access on the weekend)
  • restore them Monday at 6:00AM

# Emergency Shutdown

If you have a UPS for your hosts, and lose power, you may have a limited amount of time to shut down all of your VM infrastructure before the batteries run out. If you find yourself in this situation, or any other situation requiring the fast shutdown of everything, you can use the Emergency Shutdown feature.

# How to activate

On the host view, clicking on this button will trigger the Emergency Shutdown procedure:

  1. All running VMs will be suspended (think of it like "hibernate" on your laptop: the RAM will be stored in the storage repository).
  2. Only after this is complete, the host will be halted.

Here, you can see the running VMs are being suspended:

And finally, that's it. They are cleanly shut down with the RAM saved to disk to be resumed later:

Now the host is halted automatically.

# Powering back on

When the power outage is over, all you need to do is:

  1. Start your host.
  2. All your VMs can be resumed, your RAM is preserved and therefore your VMs will be in the exact same state as they were before the power outage.

# Terraform provider

Terraform is a cloud/platform agnostic tool for building, changing, and versioning infrastructure. Terraform can manage existing and popular service providers as well as custom in-house solutions (like Xen Orchestra). It can manage resources through their entire lifecycle or even manage infrastructure it didn't initially create.

We sponsored a developer to build a Xen Orchestra provider for Terraform (opens new window), so you can use it as a central point for your whole virtualized infrastructure. The source code is available on Github (opens new window), and contributions are welcome!

TIP

Don't miss our blog post series about it (opens new window), written by Dom Del Nano, the original developer of this provider!

It works with few steps:

  1. Install terraform
  2. Create a VM template for the new VM
  3. Provision the VM with terraform and its XO provider

From there, you can even manage your existing resources with Terraform!

# Netbox

Synchronize your pools, VMs, network interfaces and IP addresses with your Netbox (opens new window) instance.

# Netbox side

  • Go to your Netbox interface
  • Configure prefixes:
    • Go to IPAM > Prefixes > Add
    • Manually create as many prefixes as needed for your infrastructure's IP addresses

WARNING

XO will try to find the right prefix for each IP address. If it can't find a prefix that fits, the IP address won't be synchronized.

  • Create permissions:
    • Go to Admin > Permissions > Add and create 2 permissions:
      • "XO read" with action "Can view" enabled and object types:
        • Extras > custom field
        • IPAM > prefix
      • "XO read-write" with all 4 actions enabled and object types:
        • DCIM > platform
        • Extras > tag
        • IPAM > IP address
        • Tenancy > tenant (if you want to synchronize XO users with Netbox tenants)
        • Virtualization > cluster
        • Virtualization > cluster type
        • Virtualization > virtual machine
        • Virtualization > interface

  • Create a Netbox user:
    • Go to Admin > Users > Add
    • Choose a username and a password
    • Scroll down to Permissions and select the 2 permissions "XO read" and "XO read-write"
  • Create an API token:
    • Got to Admin > API Tokens > Add
    • Select the user you just created
    • Copy the token for the next step
    • Make sure "Write enabled" is checked and create it

WARNING

For testing purposes, you can create an API token bound to a Netbox superuser account, but once in production, it is highly recommended to create a dedicated user with only the required permissions.

  • Create a UUID custom field:
    • Go to Customization > Custom Fields > Add
    • Select object types:
      • Tenancy > tenant (if you want to synchronize XO users with Netbox tenants)
      • Virtualization > cluster
      • Virtualization > virtual machine
      • Virtualization > interface
    • Name it "uuid" (lower case!)

TIP

In Netbox 2.x, custom fields can be created from the Admin panel > Custom fields > Add custom field.

# In Xen Orchestra

  • Go to Xen Orchestra > Settings > Plugins > Netbox and fill out the configuration:
    • Endpoint: the URL of your Netbox instance (e.g.: https://netbox.company.net)
    • Unauthorized certificate: only for HTTPS, enable this option if your Netbox instance uses a self-signed SSL certificate
    • Token: the token you generated earlier
    • Pools: the pools you wish to automatically synchronize with Netbox
    • Synchronize users: enable this if you wish to synchronize XO users with Netbox tenants. Tenants will be assigned to the VMs the XO user created within XO. Important: if you want to enable this feature, you also need to assign the custom field "uuid" that you created in the previous step to the type "Tenancy > tenant".
    • Interval: the time interval (in hours) between 2 auto-synchronizations. Leave empty if you don't want to synchronize automatically.
  • Load the plugin (button next to the plugin's name)
  • Manual synchronization: if you correctly configured and loaded the plugin, a "Synchronize with Netbox" button will appear in every pool's Advanced tab, which allows you to manually synchronize it with Netbox

TIP

If you get a 403 Forbidden error when testing the plugin, make sure you correctly configured the "Allowed IPs" for the token you are using.

# Recipes

TIP

TODO: this section is still a work in progress!

← VM Load Balancing →