Continuous Replication 5.9 fail to delete VDI from VM XO_DELTA_EXPORT
-
Hi to all,
I recently upgraded XO to version 5.9.
The upgrade was made by setting up a new VM and compiling ver 5.9 from scratch and copying backup jobs settings from XO ver 5.6.4.One of the jobs is a "Continuous Replication" of 4 VM from one server to another, each with its own internal data storage and on the same LAN with Xen v7.1.
After the execution of the first replication job, the XO fail when try to delete the VDI VM XO_DELTA_EXPORT.In the first execution all seems work fine. The VM are replicated on the destination server and on the source server I can see the XO_DELTA_EXPORT snapshot.
But from the second run, xo-server report a problem in the log for each VM:Jun 20 14:00:17 ubuntu xo-server[1247]: cannot delete VDI AML ERP (from VM XO_DELTA_EXPORT: Local storage (352aa767-575f-cf7f-3757-b596edfa47ea))
In the storage tab of XenCenter I can see the VDIs of the VMs undeleted.
To workaround the problem I downgrade to the XO v5.6.4 that work fine.
Any idea?
Fulvio.
-
You probably should provide some additional details, such as:
- Exact XO version (latest are
xo-server 5.9.4
andxo-web 5.9.1
) - The full error log (look under Settings > Logs. Click the "eyeball" icon to get a popup window with the full details)
- Exact XO version (latest are
-
Hi Danp,
xo-server 5.9.3
xo-web 5.9.0no log in settings > Logs
in Backup > Overview > Logs the job is marked as "Finished" in green.
in /var/log/syslog:
Jun 20 14:00:00 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:00 GMT xo:xapi Snapshotting VM ACU Cobol Utility - 192.168.1.233
Jun 20 14:00:00 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:00 GMT xo:xapi Snapshotting VM AML ERP - 192.168.1.15
Jun 20 14:00:05 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:05 GMT xo:xapi task created: VDI Export (AML ERP)
Jun 20 14:00:05 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:05 GMT xo:xapi exporting VDI AML ERP (from base AML ERP)
Jun 20 14:00:07 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:07 GMT xo:xapi Creating VM AML ERP - 192.168.1.15 (2017-06-20)
Jun 20 14:00:07 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:07 GMT xo:xapi Cloning VDI AML ERP
Jun 20 14:00:10 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:10 GMT xo:xapi Creating VBD for VDI AML ERP on VM AML ERP - 192.168.1.15 (2017-06-20)
Jun 20 14:00:10 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:10 GMT xo:xapi Creating VIF for VM AML ERP - 192.168.1.15 (2017-06-20) on network Pool-wide network associated with eth0
Jun 20 14:00:10 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:10 GMT xo:xapi task created: VDI Content Import (AML ERP)
Jun 20 14:00:13 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:13 GMT xo:xapi task created: VDI Export (ACU Cobol Utility)
Jun 20 14:00:13 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:13 GMT xo:xapi exporting VDI ACU Cobol Utility (from base ACU Cobol Utility)
Jun 20 14:00:15 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:15 GMT xo:perf blocked for 13ms
Jun 20 14:00:17 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:17 GMT xo:xapi Creating VM ACU Cobol Utility - 192.168.1.233 (2017-06-20)
Jun 20 14:00:17 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:17 GMT xo:xapi Cloning VDI ACU Cobol Utility
Jun 20 14:00:17 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:17 GMT xo:xapi Deleting VM AML ERP - 192.168.1.15 (2017-06-20)
Jun 20 14:00:17 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:17 GMT xo:xapi Deleting VDI AML ERP
Jun 20 14:00:17 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:17 GMT xo:xapi task destroyed: VDI Content Import (AML ERP)
Jun 20 14:00:17 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:17 GMT xo:xapi Deleting VM XO_DELTA_EXPORT: Local storage (352aa767-575f-cf7f-3757-b596edfa47ea)
Jun 20 14:00:17 ubuntu xo-server[1247]: cannot delete VDI AML ERP (from VM XO_DELTA_EXPORT: Local storage (352aa767-575f-cf7f-3757-b596edfa47ea))
Jun 20 14:00:17 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:17 GMT xo:xapi Snapshotting VM Windows Server 2016 - 192.168.1.5
Jun 20 14:00:20 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:20 GMT xo:xapi Creating VBD for VDI ACU Cobol Utility on VM ACU Cobol Utility - 192.168.1.233 (2017-06-20)
Jun 20 14:00:20 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:20 GMT xo:xapi Creating VIF for VM ACU Cobol Utility - 192.168.1.233 (2017-06-20) on network Pool-wide network associated with eth0
Jun 20 14:00:20 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:20 GMT xo:xapi task created: VDI Content Import (ACU Cobol Utility)
Jun 20 14:00:22 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:22 GMT xo:xapi task created: VDI Export (Windows Server 2016)
Jun 20 14:00:22 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:22 GMT xo:xapi exporting VDI Windows Server 2016 (from base Windows Server 2016)
Jun 20 14:00:23 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:23 GMT xo:xapi task destroyed: VDI Export (AML ERP)
Jun 20 14:00:24 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:24 GMT xo:xapi Deleting VM ACU Cobol Utility - 192.168.1.233 (2017-06-20)
Jun 20 14:00:24 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:24 GMT xo:xapi Deleting VDI ACU Cobol Utility
Jun 20 14:00:24 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:24 GMT xo:xapi task destroyed: VDI Content Import (ACU Cobol Utility)
Jun 20 14:00:24 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:24 GMT xo:xapi Deleting VM XO_DELTA_EXPORT: Local storage (352aa767-575f-cf7f-3757-b596edfa47ea)
Jun 20 14:00:24 ubuntu xo-server[1247]: cannot delete VDI ACU Cobol Utility (from VM XO_DELTA_EXPORT: Local storage (352aa767-575f-cf7f-3757-b596edfa47ea))
Jun 20 14:00:24 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:24 GMT xo:xapi Snapshotting VM CDT ERP - 192.168.1.10
Jun 20 14:00:25 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:25 GMT xo:xapi Creating VM Windows Server 2016 - 192.168.1.5 (2017-06-20)
Jun 20 14:00:25 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:25 GMT xo:xapi Cloning VDI Windows Server 2016
Jun 20 14:00:29 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:29 GMT xo:xapi task destroyed: VDI Export (ACU Cobol Utility)
Jun 20 14:00:29 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:29 GMT xo:xapi task created: VDI Export (CDT ERP)
Jun 20 14:00:29 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:29 GMT xo:xapi exporting VDI CDT ERP (from base CDT ERP)
Jun 20 14:00:31 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:31 GMT xo:xapi Creating VM CDT ERP - 192.168.1.10 (2017-06-20)
Jun 20 14:00:31 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:31 GMT xo:xapi Cloning VDI CDT ERP
Jun 20 14:00:35 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:35 GMT xo:xapi Creating VBD for VDI CDT ERP on VM CDT ERP - 192.168.1.10 (2017-06-20)
Jun 20 14:00:35 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:35 GMT xo:xapi Creating VIF for VM CDT ERP - 192.168.1.10 (2017-06-20) on network Pool-wide network associated with eth0
Jun 20 14:00:35 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:35 GMT xo:xapi task created: VDI Content Import (CDT ERP)
Jun 20 14:00:43 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:43 GMT xo:xapi Creating VBD for VDI Windows Server 2016 on VM Windows Server 2016 - 192.168.1.5 (2017-06-20)
Jun 20 14:00:43 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:43 GMT xo:xapi Creating VIF for VM Windows Server 2016 - 192.168.1.5 (2017-06-20) on network Pool-wide network associated with eth0
Jun 20 14:00:43 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:43 GMT xo:xapi task created: VDI Content Import (Windows Server 2016)
Jun 20 14:00:49 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:49 GMT xo:xapi Deleting VM Windows Server 2016 - 192.168.1.5 (2017-06-20)
Jun 20 14:00:49 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:49 GMT xo:xapi Deleting VDI Windows Server 2016
Jun 20 14:00:49 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:49 GMT xo:xapi task destroyed: VDI Content Import (Windows Server 2016)
Jun 20 14:00:49 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:49 GMT xo:xapi Deleting VM XO_DELTA_EXPORT: Local storage (352aa767-575f-cf7f-3757-b596edfa47ea)
Jun 20 14:00:49 ubuntu xo-server[1247]: cannot delete VDI Windows Server 2016 (from VM XO_DELTA_EXPORT: Local storage (352aa767-575f-cf7f-3757-b596edfa47ea))
Jun 20 14:00:51 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:51 GMT xo:xapi task destroyed: VDI Export (Windows Server 2016)
Jun 20 14:01:00 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:01:00 GMT xo:xapi Deleting VM CDT ERP - 192.168.1.10 (2017-06-20)
Jun 20 14:01:00 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:01:00 GMT xo:xapi Deleting VDI CDT ERP
Jun 20 14:01:00 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:01:00 GMT xo:xapi task destroyed: VDI Content Import (CDT ERP)
Jun 20 14:01:00 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:01:00 GMT xo:xapi Deleting VM XO_DELTA_EXPORT: Local storage (352aa767-575f-cf7f-3757-b596edfa47ea)
Jun 20 14:01:00 ubuntu xo-server[1247]: cannot delete VDI CDT ERP (from VM XO_DELTA_EXPORT: Local storage (352aa767-575f-cf7f-3757-b596edfa47ea))
Jun 20 14:01:02 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:01:02 GMT xo:xapi task destroyed: VDI Export (CDT ERP)
-
@fulvio There is a later release, so you could try updating to that to see if the issue is resolved or continues to occur.
Have you encountered this with any other non-CR jobs?
A lot has changed since XO v5.6.4. Any ideas @olivierlambert?
-
no, no other issue...
we use also the normal backup without any problem.I try to upgrade to the latest release and I will inform you here.
Thanks.
-
@Danp @olivierlambert This morning I had try with the latest release (
xo-server 5.9.4
-xo-web 5.9.1
) but the issue still persist.Jun 21 12:40:52 ubuntu xo-server[1574]: cannot delete VDI AML ERP (from VM XO_DELTA_EXPORT: Local storage (352aa767-575f-cf7f-3757-b596edfa47ea))
-
What snapshots exist after the job finishes? Are there more than one per VM?
You may want to post your issue to their official issues list over on Github -- https://github.com/vatesfr/xo-web/issues/new
-
@Danp at the end of the job only the last new snapshot remain, and only one for each VM (the last)
In the storage repo we can see some vdi zombie.
-
Are you able to manually remove these remnant VDIs?
Are they listed in the "Orphaned snapshot VDIs" under Dashboard > Health?
I see you opened an issue on GH. Hopefully one of the devs will chime in soon.
-
@Danp Yes, from Xencenter I can delete them.
No orphaned listed inside Health section.Meanwhile I continue to use v5.6.4 that works fine
-
@fulvio can you follow instructions on the issue? I think we fixed that
https://github.com/vatesfr/xo-web/issues/2227
-
@olivierlambert @Danp I'm sorry Olivier, but I don't know which are the right steps to apply this patch (I dont know GitHub very well)
-
You need to go into the
xo-server
folder and switch to the right branch (git checkout fix-export-import
thenyarn
)
-
@olivierlambert ok... i'm gonna try. Thanks.
-
root@ubuntu:/opt/xo-server# git checkout fix-export-import
error: pathspec 'fix-export-import' did not match any file(s) known to git.
-
git pull
first to get the latest data from the remote, thengit checkout
the branch.
-
root@ubuntu:/opt/xo-server# git pull remote: Counting objects: 150, done. remote: Compressing objects: 100% (58/58), done. remote: Total 150 (delta 101), reused 127 (delta 92), pack-reused 0 Ricezione degli oggetti: 100% (150/150), 119.97 KiB | 0 bytes/s, done. Risoluzione dei delta: 100% (101/101), completed with 32 local objects. Da https://github.com/vatesfr/xo-server 2d39a1a..dc9af66 backupRestoration -> origin/backupRestoration edbeb55..eac4178 cbs-exposeVmAppliance -> origin/cbs-exposeVmAppliance * [nuovo branch] fix-export-import -> origin/fix-export-import * [nuovo branch] fixCreateVm-resourceSet -> origin/fixCreateVm-resourceSet de7672b..ff16533 next-release -> origin/next-release f217c84..9ac1545 nr-xosan2 -> origin/nr-xosan2 Already up-to-date. root@ubuntu:/opt/xo-server# root@ubuntu:/opt/xo-server# git checkout fix-export-import -bash: root@ubuntu:/opt/xo-server#: File o directory non esistente
-
sorry... (copy and paste error...)
I redo all but from the "git checkout" command I receive a warning...root@ubuntu:/opt/xo-server# git pull remote: Counting objects: 150, done. remote: Compressing objects: 100% (58/58), done. remote: Total 150 (delta 101), reused 127 (delta 92), pack-reused 0 Ricezione degli oggetti: 100% (150/150), 119.97 KiB | 0 bytes/s, done. Risoluzione dei delta: 100% (101/101), completed with 32 local objects. Da https://github.com/vatesfr/xo-server 2d39a1a..dc9af66 backupRestoration -> origin/backupRestoration edbeb55..eac4178 cbs-exposeVmAppliance -> origin/cbs-exposeVmAppliance * [nuovo branch] fix-export-import -> origin/fix-export-import * [nuovo branch] fixCreateVm-resourceSet -> origin/fixCreateVm-resourceSet de7672b..ff16533 next-release -> origin/next-release f217c84..9ac1545 nr-xosan2 -> origin/nr-xosan2 Already up-to-date. root@ubuntu:/opt/xo-server# root@ubuntu:/opt/xo-server# git branch -a * stable remotes/origin/HEAD -> origin/next-release remotes/origin/architecture-rework remotes/origin/babel-config remotes/origin/backup-cleanup remotes/origin/backupRestoration remotes/origin/cancelable-backups remotes/origin/cancellable-vm-import remotes/origin/cbs-exposeVmAppliance remotes/origin/collection-patch remotes/origin/disable-quiesce-fallback remotes/origin/extract-xo-fs-to-module remotes/origin/fix-export-import remotes/origin/fix-registerUser remotes/origin/fixCreateVm-resourceSet remotes/origin/fixResourceSet remotes/origin/force-ext4 remotes/origin/jobTimezone remotes/origin/master remotes/origin/next-release remotes/origin/nr-try-fix-xosan-beta remotes/origin/nr-update-noVNC remotes/origin/nr-xosan remotes/origin/nr-xosan2 remotes/origin/olt-detectUnheathlyChain remotes/origin/olt-testsrstats remotes/origin/patch remotes/origin/revert-CR-DR-job remotes/origin/stable remotes/origin/utils remotes/origin/why-is-node-running root@ubuntu:/opt/xo-server# git checkout fix-export-import error: Your local changes to the following files would be overwritten by checkout: package.json Please commit your changes or stash them before you switch branches. Aborting root@ubuntu:/opt/xo-server#
-
Well your git is not doing this automatically, that's why it's hard to support non-XOA installs ^^
So do a:
git checkout origin/fix-export-import
-
same message:
root@ubuntu:/opt/xo-server# git checkout origin/fix-export-import error: Your local changes to the following files would be overwritten by checkout: package.json Please commit your changes or stash them before you switch branches. Aborting root@ubuntu:/opt/xo-server#