Continuous Replication 5.9 fail to delete VDI from VM XO_DELTA_EXPORT

[fulvio]

Hi to all,

I recently upgraded XO to version 5.9.
The upgrade was made by setting up a new VM and compiling ver 5.9 from scratch and copying backup jobs settings from XO ver 5.6.4.

One of the jobs is a "Continuous Replication" of 4 VM from one server to another, each with its own internal data storage and on the same LAN with Xen v7.1.
After the execution of the first replication job, the XO fail when try to delete the VDI VM XO_DELTA_EXPORT.

In the first execution all seems work fine. The VM are replicated on the destination server and on the source server I can see the XO_DELTA_EXPORT snapshot.
But from the second run, xo-server report a problem in the log for each VM:

Jun 20 14:00:17 ubuntu xo-server[1247]: cannot delete VDI AML ERP (from VM XO_DELTA_EXPORT: Local storage (352aa767-575f-cf7f-3757-b596edfa47ea))

In the storage tab of XenCenter I can see the VDIs of the VMs undeleted.

To workaround the problem I downgrade to the XO v5.6.4 that work fine.

Any idea?

Fulvio.

[Danp]

You probably should provide some additional details, such as:

• Exact XO version (latest are xo-server 5.9.4 and xo-web 5.9.1)
• The full error log (look under Settings > Logs. Click the "eyeball" icon to get a popup window with the full details)

[fulvio]

Hi Danp,

xo-server 5.9.3
xo-web 5.9.0

no log in settings > Logs

in Backup > Overview > Logs the job is marked as "Finished" in green.

in /var/log/syslog:

Jun 20 14:00:00 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:00 GMT xo:xapi Snapshotting VM ACU Cobol Utility - 192.168.1.233
Jun 20 14:00:00 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:00 GMT xo:xapi Snapshotting VM AML ERP - 192.168.1.15
Jun 20 14:00:05 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:05 GMT xo:xapi task created: VDI Export (AML ERP)
Jun 20 14:00:05 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:05 GMT xo:xapi exporting VDI AML ERP (from base AML ERP)
Jun 20 14:00:07 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:07 GMT xo:xapi Creating VM AML ERP - 192.168.1.15 (2017-06-20)
Jun 20 14:00:07 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:07 GMT xo:xapi Cloning VDI AML ERP
Jun 20 14:00:10 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:10 GMT xo:xapi Creating VBD for VDI AML ERP on VM AML ERP - 192.168.1.15 (2017-06-20)
Jun 20 14:00:10 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:10 GMT xo:xapi Creating VIF for VM AML ERP - 192.168.1.15 (2017-06-20) on network Pool-wide network associated with eth0
Jun 20 14:00:10 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:10 GMT xo:xapi task created: VDI Content Import (AML ERP)
Jun 20 14:00:13 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:13 GMT xo:xapi task created: VDI Export (ACU Cobol Utility)
Jun 20 14:00:13 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:13 GMT xo:xapi exporting VDI ACU Cobol Utility (from base ACU Cobol Utility)
Jun 20 14:00:15 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:15 GMT xo:perf blocked for 13ms
Jun 20 14:00:17 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:17 GMT xo:xapi Creating VM ACU Cobol Utility - 192.168.1.233 (2017-06-20)
Jun 20 14:00:17 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:17 GMT xo:xapi Cloning VDI ACU Cobol Utility
Jun 20 14:00:17 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:17 GMT xo:xapi Deleting VM AML ERP - 192.168.1.15 (2017-06-20)
Jun 20 14:00:17 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:17 GMT xo:xapi Deleting VDI AML ERP
Jun 20 14:00:17 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:17 GMT xo:xapi task destroyed: VDI Content Import (AML ERP)
Jun 20 14:00:17 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:17 GMT xo:xapi Deleting VM XO_DELTA_EXPORT: Local storage (352aa767-575f-cf7f-3757-b596edfa47ea)
Jun 20 14:00:17 ubuntu xo-server[1247]: cannot delete VDI AML ERP (from VM XO_DELTA_EXPORT: Local storage (352aa767-575f-cf7f-3757-b596edfa47ea))
Jun 20 14:00:17 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:17 GMT xo:xapi Snapshotting VM Windows Server 2016 - 192.168.1.5
Jun 20 14:00:20 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:20 GMT xo:xapi Creating VBD for VDI ACU Cobol Utility on VM ACU Cobol Utility - 192.168.1.233 (2017-06-20)
Jun 20 14:00:20 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:20 GMT xo:xapi Creating VIF for VM ACU Cobol Utility - 192.168.1.233 (2017-06-20) on network Pool-wide network associated with eth0
Jun 20 14:00:20 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:20 GMT xo:xapi task created: VDI Content Import (ACU Cobol Utility)
Jun 20 14:00:22 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:22 GMT xo:xapi task created: VDI Export (Windows Server 2016)
Jun 20 14:00:22 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:22 GMT xo:xapi exporting VDI Windows Server 2016 (from base Windows Server 2016)
Jun 20 14:00:23 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:23 GMT xo:xapi task destroyed: VDI Export (AML ERP)
Jun 20 14:00:24 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:24 GMT xo:xapi Deleting VM ACU Cobol Utility - 192.168.1.233 (2017-06-20)
Jun 20 14:00:24 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:24 GMT xo:xapi Deleting VDI ACU Cobol Utility
Jun 20 14:00:24 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:24 GMT xo:xapi task destroyed: VDI Content Import (ACU Cobol Utility)
Jun 20 14:00:24 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:24 GMT xo:xapi Deleting VM XO_DELTA_EXPORT: Local storage (352aa767-575f-cf7f-3757-b596edfa47ea)
Jun 20 14:00:24 ubuntu xo-server[1247]: cannot delete VDI ACU Cobol Utility (from VM XO_DELTA_EXPORT: Local storage (352aa767-575f-cf7f-3757-b596edfa47ea))
Jun 20 14:00:24 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:24 GMT xo:xapi Snapshotting VM CDT ERP - 192.168.1.10
Jun 20 14:00:25 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:25 GMT xo:xapi Creating VM Windows Server 2016 - 192.168.1.5 (2017-06-20)
Jun 20 14:00:25 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:25 GMT xo:xapi Cloning VDI Windows Server 2016
Jun 20 14:00:29 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:29 GMT xo:xapi task destroyed: VDI Export (ACU Cobol Utility)
Jun 20 14:00:29 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:29 GMT xo:xapi task created: VDI Export (CDT ERP)
Jun 20 14:00:29 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:29 GMT xo:xapi exporting VDI CDT ERP (from base CDT ERP)
Jun 20 14:00:31 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:31 GMT xo:xapi Creating VM CDT ERP - 192.168.1.10 (2017-06-20)
Jun 20 14:00:31 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:31 GMT xo:xapi Cloning VDI CDT ERP
Jun 20 14:00:35 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:35 GMT xo:xapi Creating VBD for VDI CDT ERP on VM CDT ERP - 192.168.1.10 (2017-06-20)
Jun 20 14:00:35 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:35 GMT xo:xapi Creating VIF for VM CDT ERP - 192.168.1.10 (2017-06-20) on network Pool-wide network associated with eth0
Jun 20 14:00:35 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:35 GMT xo:xapi task created: VDI Content Import (CDT ERP)
Jun 20 14:00:43 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:43 GMT xo:xapi Creating VBD for VDI Windows Server 2016 on VM Windows Server 2016 - 192.168.1.5 (2017-06-20)
Jun 20 14:00:43 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:43 GMT xo:xapi Creating VIF for VM Windows Server 2016 - 192.168.1.5 (2017-06-20) on network Pool-wide network associated with eth0
Jun 20 14:00:43 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:43 GMT xo:xapi task created: VDI Content Import (Windows Server 2016)
Jun 20 14:00:49 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:49 GMT xo:xapi Deleting VM Windows Server 2016 - 192.168.1.5 (2017-06-20)
Jun 20 14:00:49 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:49 GMT xo:xapi Deleting VDI Windows Server 2016
Jun 20 14:00:49 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:49 GMT xo:xapi task destroyed: VDI Content Import (Windows Server 2016)
Jun 20 14:00:49 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:49 GMT xo:xapi Deleting VM XO_DELTA_EXPORT: Local storage (352aa767-575f-cf7f-3757-b596edfa47ea)
Jun 20 14:00:49 ubuntu xo-server[1247]: cannot delete VDI Windows Server 2016 (from VM XO_DELTA_EXPORT: Local storage (352aa767-575f-cf7f-3757-b596edfa47ea))
Jun 20 14:00:51 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:51 GMT xo:xapi task destroyed: VDI Export (Windows Server 2016)
Jun 20 14:01:00 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:01:00 GMT xo:xapi Deleting VM CDT ERP - 192.168.1.10 (2017-06-20)
Jun 20 14:01:00 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:01:00 GMT xo:xapi Deleting VDI CDT ERP
Jun 20 14:01:00 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:01:00 GMT xo:xapi task destroyed: VDI Content Import (CDT ERP)
Jun 20 14:01:00 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:01:00 GMT xo:xapi Deleting VM XO_DELTA_EXPORT: Local storage (352aa767-575f-cf7f-3757-b596edfa47ea)
Jun 20 14:01:00 ubuntu xo-server[1247]: cannot delete VDI CDT ERP (from VM XO_DELTA_EXPORT: Local storage (352aa767-575f-cf7f-3757-b596edfa47ea))
Jun 20 14:01:02 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:01:02 GMT xo:xapi task destroyed: VDI Export (CDT ERP)

[Danp]

@fulvio There is a later release, so you could try updating to that to see if the issue is resolved or continues to occur.

Have you encountered this with any other non-CR jobs?

A lot has changed since XO v5.6.4. Any ideas @olivierlambert?

[fulvio]

no, no other issue...
we use also the normal backup without any problem.

I try to upgrade to the latest release and I will inform you here.

Thanks.

[fulvio]

@Danp @olivierlambert This morning I had try with the latest release (xo-server 5.9.4 - xo-web 5.9.1) but the issue still persist.

Jun 21 12:40:52 ubuntu xo-server[1574]: cannot delete VDI AML ERP (from VM XO_DELTA_EXPORT: Local storage (352aa767-575f-cf7f-3757-b596edfa47ea))

[Danp]

What snapshots exist after the job finishes? Are there more than one per VM?

You may want to post your issue to their official issues list over on Github -- https://github.com/vatesfr/xo-web/issues/new

[fulvio]

@Danp at the end of the job only the last new snapshot remain, and only one for each VM (the last)

In the storage repo we can see some vdi zombie.

[Danp]

Are you able to manually remove these remnant VDIs?

Are they listed in the "Orphaned snapshot VDIs" under Dashboard > Health?

I see you opened an issue on GH. Hopefully one of the devs will chime in soon.

[fulvio]

@Danp Yes, from Xencenter I can delete them.
No orphaned listed inside Health section.

Meanwhile I continue to use v5.6.4 that works fine

[olivierlambert]

@fulvio can you follow instructions on the issue? I think we fixed that https://github.com/vatesfr/xo-web/issues/2227

fulvio61 created this issue in vatesfr/xo-web

closed Continuous Replication 5.9 fail to delete VDI from VM XO_DELTA_EXPORT #2227

[fulvio]

@olivierlambert @Danp I'm sorry Olivier, but I don't know which are the right steps to apply this patch (I dont know GitHub very well)

[olivierlambert]

You need to go into the xo-server folder and switch to the right branch (git checkout fix-export-import then yarn)

[fulvio]

@olivierlambert ok... i'm gonna try. Thanks.

[fulvio]

@olivierlambert

root@ubuntu:/opt/xo-server# git checkout fix-export-import
error: pathspec 'fix-export-import' did not match any file(s) known to git.

[olivierlambert]

git pull first to get the latest data from the remote, then git checkout the branch.

[fulvio]

@olivierlambert

root@ubuntu:/opt/xo-server# git pull
remote: Counting objects: 150, done.
remote: Compressing objects: 100% (58/58), done.
remote: Total 150 (delta 101), reused 127 (delta 92), pack-reused 0
Ricezione degli oggetti: 100% (150/150), 119.97 KiB | 0 bytes/s, done.
Risoluzione dei delta: 100% (101/101), completed with 32 local objects.
Da https://github.com/vatesfr/xo-server
   2d39a1a..dc9af66  backupRestoration -> origin/backupRestoration
   edbeb55..eac4178  cbs-exposeVmAppliance -> origin/cbs-exposeVmAppliance
 * [nuovo branch]    fix-export-import -> origin/fix-export-import
 * [nuovo branch]    fixCreateVm-resourceSet -> origin/fixCreateVm-resourceSet
   de7672b..ff16533  next-release -> origin/next-release
   f217c84..9ac1545  nr-xosan2  -> origin/nr-xosan2
Already up-to-date.

root@ubuntu:/opt/xo-server# root@ubuntu:/opt/xo-server# git checkout fix-export-import
-bash: root@ubuntu:/opt/xo-server#: File o directory non esistente

[fulvio]

@olivierlambert

sorry... (copy and paste error...)
I redo all but from the "git checkout" command I receive a warning...

root@ubuntu:/opt/xo-server# git pull
remote: Counting objects: 150, done.
remote: Compressing objects: 100% (58/58), done.
remote: Total 150 (delta 101), reused 127 (delta 92), pack-reused 0
Ricezione degli oggetti: 100% (150/150), 119.97 KiB | 0 bytes/s, done.
Risoluzione dei delta: 100% (101/101), completed with 32 local objects.
Da https://github.com/vatesfr/xo-server
   2d39a1a..dc9af66  backupRestoration -> origin/backupRestoration
   edbeb55..eac4178  cbs-exposeVmAppliance -> origin/cbs-exposeVmAppliance
 * [nuovo branch]    fix-export-import -> origin/fix-export-import
 * [nuovo branch]    fixCreateVm-resourceSet -> origin/fixCreateVm-resourceSet
   de7672b..ff16533  next-release -> origin/next-release
   f217c84..9ac1545  nr-xosan2  -> origin/nr-xosan2
Already up-to-date.
root@ubuntu:/opt/xo-server#

root@ubuntu:/opt/xo-server# git branch -a
* stable
  remotes/origin/HEAD -> origin/next-release
  remotes/origin/architecture-rework
  remotes/origin/babel-config
  remotes/origin/backup-cleanup
  remotes/origin/backupRestoration
  remotes/origin/cancelable-backups
  remotes/origin/cancellable-vm-import
  remotes/origin/cbs-exposeVmAppliance
  remotes/origin/collection-patch
  remotes/origin/disable-quiesce-fallback
  remotes/origin/extract-xo-fs-to-module
  remotes/origin/fix-export-import
  remotes/origin/fix-registerUser
  remotes/origin/fixCreateVm-resourceSet
  remotes/origin/fixResourceSet
  remotes/origin/force-ext4
  remotes/origin/jobTimezone
  remotes/origin/master
  remotes/origin/next-release
  remotes/origin/nr-try-fix-xosan-beta
  remotes/origin/nr-update-noVNC
  remotes/origin/nr-xosan
  remotes/origin/nr-xosan2
  remotes/origin/olt-detectUnheathlyChain
  remotes/origin/olt-testsrstats
  remotes/origin/patch
  remotes/origin/revert-CR-DR-job
  remotes/origin/stable
  remotes/origin/utils
  remotes/origin/why-is-node-running


root@ubuntu:/opt/xo-server# git checkout fix-export-import
error: Your local changes to the following files would be overwritten by checkout:
        package.json
Please commit your changes or stash them before you switch branches.
Aborting
root@ubuntu:/opt/xo-server#

[olivierlambert]

Well your git is not doing this automatically, that's why it's hard to support non-XOA installs ^^

So do a:
git checkout origin/fix-export-import

[fulvio]

@olivierlambert

same message:

root@ubuntu:/opt/xo-server# git checkout origin/fix-export-import
error: Your local changes to the following files would be overwritten by checkout:
        package.json
Please commit your changes or stash them before you switch branches.
Aborting
root@ubuntu:/opt/xo-server#