Continuous Replication 5.9 fail to delete VDI from VM XO_DELTA_EXPORT



  • Hi to all,

    I recently upgraded XO to version 5.9.
    The upgrade was made by setting up a new VM and compiling ver 5.9 from scratch and copying backup jobs settings from XO ver 5.6.4.

    One of the jobs is a "Continuous Replication" of 4 VM from one server to another, each with its own internal data storage and on the same LAN with Xen v7.1.
    After the execution of the first replication job, the XO fail when try to delete the VDI VM XO_DELTA_EXPORT.

    In the first execution all seems work fine. The VM are replicated on the destination server and on the source server I can see the XO_DELTA_EXPORT snapshot.
    But from the second run, xo-server report a problem in the log for each VM:

    Jun 20 14:00:17 ubuntu xo-server[1247]: cannot delete VDI AML ERP (from VM XO_DELTA_EXPORT: Local storage (352aa767-575f-cf7f-3757-b596edfa47ea))

    In the storage tab of XenCenter I can see the VDIs of the VMs undeleted.

    To workaround the problem I downgrade to the XO v5.6.4 that work fine.

    Any idea?

    Fulvio.



  • You probably should provide some additional details, such as:

    • Exact XO version (latest are xo-server 5.9.4 and xo-web 5.9.1)
    • The full error log (look under Settings > Logs. Click the "eyeball" icon to get a popup window with the full details)


  • Hi Danp,

    xo-server 5.9.3
    xo-web 5.9.0

    no log in settings > Logs

    in Backup > Overview > Logs the job is marked as "Finished" in green.

    in /var/log/syslog:

    Jun 20 14:00:00 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:00 GMT xo:xapi Snapshotting VM ACU Cobol Utility - 192.168.1.233
    Jun 20 14:00:00 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:00 GMT xo:xapi Snapshotting VM AML ERP - 192.168.1.15
    Jun 20 14:00:05 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:05 GMT xo:xapi task created: VDI Export (AML ERP)
    Jun 20 14:00:05 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:05 GMT xo:xapi exporting VDI AML ERP (from base AML ERP)
    Jun 20 14:00:07 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:07 GMT xo:xapi Creating VM AML ERP - 192.168.1.15 (2017-06-20)
    Jun 20 14:00:07 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:07 GMT xo:xapi Cloning VDI AML ERP
    Jun 20 14:00:10 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:10 GMT xo:xapi Creating VBD for VDI AML ERP on VM AML ERP - 192.168.1.15 (2017-06-20)
    Jun 20 14:00:10 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:10 GMT xo:xapi Creating VIF for VM AML ERP - 192.168.1.15 (2017-06-20) on network Pool-wide network associated with eth0
    Jun 20 14:00:10 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:10 GMT xo:xapi task created: VDI Content Import (AML ERP)
    Jun 20 14:00:13 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:13 GMT xo:xapi task created: VDI Export (ACU Cobol Utility)
    Jun 20 14:00:13 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:13 GMT xo:xapi exporting VDI ACU Cobol Utility (from base ACU Cobol Utility)
    Jun 20 14:00:15 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:15 GMT xo:perf blocked for 13ms
    Jun 20 14:00:17 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:17 GMT xo:xapi Creating VM ACU Cobol Utility - 192.168.1.233 (2017-06-20)
    Jun 20 14:00:17 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:17 GMT xo:xapi Cloning VDI ACU Cobol Utility
    Jun 20 14:00:17 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:17 GMT xo:xapi Deleting VM AML ERP - 192.168.1.15 (2017-06-20)
    Jun 20 14:00:17 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:17 GMT xo:xapi Deleting VDI AML ERP
    Jun 20 14:00:17 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:17 GMT xo:xapi task destroyed: VDI Content Import (AML ERP)
    Jun 20 14:00:17 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:17 GMT xo:xapi Deleting VM XO_DELTA_EXPORT: Local storage (352aa767-575f-cf7f-3757-b596edfa47ea)
    Jun 20 14:00:17 ubuntu xo-server[1247]: cannot delete VDI AML ERP (from VM XO_DELTA_EXPORT: Local storage (352aa767-575f-cf7f-3757-b596edfa47ea))
    Jun 20 14:00:17 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:17 GMT xo:xapi Snapshotting VM Windows Server 2016 - 192.168.1.5
    Jun 20 14:00:20 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:20 GMT xo:xapi Creating VBD for VDI ACU Cobol Utility on VM ACU Cobol Utility - 192.168.1.233 (2017-06-20)
    Jun 20 14:00:20 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:20 GMT xo:xapi Creating VIF for VM ACU Cobol Utility - 192.168.1.233 (2017-06-20) on network Pool-wide network associated with eth0
    Jun 20 14:00:20 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:20 GMT xo:xapi task created: VDI Content Import (ACU Cobol Utility)
    Jun 20 14:00:22 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:22 GMT xo:xapi task created: VDI Export (Windows Server 2016)
    Jun 20 14:00:22 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:22 GMT xo:xapi exporting VDI Windows Server 2016 (from base Windows Server 2016)
    Jun 20 14:00:23 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:23 GMT xo:xapi task destroyed: VDI Export (AML ERP)
    Jun 20 14:00:24 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:24 GMT xo:xapi Deleting VM ACU Cobol Utility - 192.168.1.233 (2017-06-20)
    Jun 20 14:00:24 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:24 GMT xo:xapi Deleting VDI ACU Cobol Utility
    Jun 20 14:00:24 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:24 GMT xo:xapi task destroyed: VDI Content Import (ACU Cobol Utility)
    Jun 20 14:00:24 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:24 GMT xo:xapi Deleting VM XO_DELTA_EXPORT: Local storage (352aa767-575f-cf7f-3757-b596edfa47ea)
    Jun 20 14:00:24 ubuntu xo-server[1247]: cannot delete VDI ACU Cobol Utility (from VM XO_DELTA_EXPORT: Local storage (352aa767-575f-cf7f-3757-b596edfa47ea))
    Jun 20 14:00:24 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:24 GMT xo:xapi Snapshotting VM CDT ERP - 192.168.1.10
    Jun 20 14:00:25 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:25 GMT xo:xapi Creating VM Windows Server 2016 - 192.168.1.5 (2017-06-20)
    Jun 20 14:00:25 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:25 GMT xo:xapi Cloning VDI Windows Server 2016
    Jun 20 14:00:29 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:29 GMT xo:xapi task destroyed: VDI Export (ACU Cobol Utility)
    Jun 20 14:00:29 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:29 GMT xo:xapi task created: VDI Export (CDT ERP)
    Jun 20 14:00:29 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:29 GMT xo:xapi exporting VDI CDT ERP (from base CDT ERP)
    Jun 20 14:00:31 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:31 GMT xo:xapi Creating VM CDT ERP - 192.168.1.10 (2017-06-20)
    Jun 20 14:00:31 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:31 GMT xo:xapi Cloning VDI CDT ERP
    Jun 20 14:00:35 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:35 GMT xo:xapi Creating VBD for VDI CDT ERP on VM CDT ERP - 192.168.1.10 (2017-06-20)
    Jun 20 14:00:35 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:35 GMT xo:xapi Creating VIF for VM CDT ERP - 192.168.1.10 (2017-06-20) on network Pool-wide network associated with eth0
    Jun 20 14:00:35 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:35 GMT xo:xapi task created: VDI Content Import (CDT ERP)
    Jun 20 14:00:43 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:43 GMT xo:xapi Creating VBD for VDI Windows Server 2016 on VM Windows Server 2016 - 192.168.1.5 (2017-06-20)
    Jun 20 14:00:43 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:43 GMT xo:xapi Creating VIF for VM Windows Server 2016 - 192.168.1.5 (2017-06-20) on network Pool-wide network associated with eth0
    Jun 20 14:00:43 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:43 GMT xo:xapi task created: VDI Content Import (Windows Server 2016)
    Jun 20 14:00:49 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:49 GMT xo:xapi Deleting VM Windows Server 2016 - 192.168.1.5 (2017-06-20)
    Jun 20 14:00:49 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:49 GMT xo:xapi Deleting VDI Windows Server 2016
    Jun 20 14:00:49 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:49 GMT xo:xapi task destroyed: VDI Content Import (Windows Server 2016)
    Jun 20 14:00:49 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:49 GMT xo:xapi Deleting VM XO_DELTA_EXPORT: Local storage (352aa767-575f-cf7f-3757-b596edfa47ea)
    Jun 20 14:00:49 ubuntu xo-server[1247]: cannot delete VDI Windows Server 2016 (from VM XO_DELTA_EXPORT: Local storage (352aa767-575f-cf7f-3757-b596edfa47ea))
    Jun 20 14:00:51 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:00:51 GMT xo:xapi task destroyed: VDI Export (Windows Server 2016)
    Jun 20 14:01:00 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:01:00 GMT xo:xapi Deleting VM CDT ERP - 192.168.1.10 (2017-06-20)
    Jun 20 14:01:00 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:01:00 GMT xo:xapi Deleting VDI CDT ERP
    Jun 20 14:01:00 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:01:00 GMT xo:xapi task destroyed: VDI Content Import (CDT ERP)
    Jun 20 14:01:00 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:01:00 GMT xo:xapi Deleting VM XO_DELTA_EXPORT: Local storage (352aa767-575f-cf7f-3757-b596edfa47ea)
    Jun 20 14:01:00 ubuntu xo-server[1247]: cannot delete VDI CDT ERP (from VM XO_DELTA_EXPORT: Local storage (352aa767-575f-cf7f-3757-b596edfa47ea))
    Jun 20 14:01:02 ubuntu xo-server[1247]: Tue, 20 Jun 2017 12:01:02 GMT xo:xapi task destroyed: VDI Export (CDT ERP)



  • @fulvio There is a later release, so you could try updating to that to see if the issue is resolved or continues to occur.

    Have you encountered this with any other non-CR jobs?

    A lot has changed since XO v5.6.4. Any ideas @olivierlambert?



  • no, no other issue...
    we use also the normal backup without any problem.

    I try to upgrade to the latest release and I will inform you here.

    Thanks.



  • @Danp @olivierlambert This morning I had try with the latest release (xo-server 5.9.4 - xo-web 5.9.1) but the issue still persist.

    Jun 21 12:40:52 ubuntu xo-server[1574]: cannot delete VDI AML ERP (from VM XO_DELTA_EXPORT: Local storage (352aa767-575f-cf7f-3757-b596edfa47ea))



  • What snapshots exist after the job finishes? Are there more than one per VM?

    You may want to post your issue to their official issues list over on Github -- https://github.com/vatesfr/xo-web/issues/new



  • @Danp at the end of the job only the last new snapshot remain, and only one for each VM (the last)

    In the storage repo we can see some vdi zombie.

    0_1498048426403_28be12ad-411d-4952-81a5-fbe9cc1bf2bd-image.png

    0_1498048657006_36c4b871-b8c6-4f37-95ef-0f67fdd5aa64-image.png



  • Are you able to manually remove these remnant VDIs?

    Are they listed in the "Orphaned snapshot VDIs" under Dashboard > Health?

    I see you opened an issue on GH. Hopefully one of the devs will chime in soon.



  • @Danp Yes, from Xencenter I can delete them.
    No orphaned listed inside Health section.

    Meanwhile I continue to use v5.6.4 that works fine 🙂



  • @fulvio can you follow instructions on the issue? I think we fixed that 🙂 https://github.com/vatesfr/xo-web/issues/2227



  • @olivierlambert @Danp I'm sorry Olivier, but I don't know which are the right steps to apply this patch (I dont know GitHub very well)



  • You need to go into the xo-server folder and switch to the right branch (git checkout fix-export-import then yarn)



  • @olivierlambert ok... i'm gonna try. Thanks.



  • @olivierlambert

    root@ubuntu:/opt/xo-server# git checkout fix-export-import
    error: pathspec 'fix-export-import' did not match any file(s) known to git.



  • git pull first to get the latest data from the remote, then git checkout the branch.



  • @olivierlambert

    root@ubuntu:/opt/xo-server# git pull
    remote: Counting objects: 150, done.
    remote: Compressing objects: 100% (58/58), done.
    remote: Total 150 (delta 101), reused 127 (delta 92), pack-reused 0
    Ricezione degli oggetti: 100% (150/150), 119.97 KiB | 0 bytes/s, done.
    Risoluzione dei delta: 100% (101/101), completed with 32 local objects.
    Da https://github.com/vatesfr/xo-server
       2d39a1a..dc9af66  backupRestoration -> origin/backupRestoration
       edbeb55..eac4178  cbs-exposeVmAppliance -> origin/cbs-exposeVmAppliance
     * [nuovo branch]    fix-export-import -> origin/fix-export-import
     * [nuovo branch]    fixCreateVm-resourceSet -> origin/fixCreateVm-resourceSet
       de7672b..ff16533  next-release -> origin/next-release
       f217c84..9ac1545  nr-xosan2  -> origin/nr-xosan2
    Already up-to-date.
    
    root@ubuntu:/opt/xo-server# root@ubuntu:/opt/xo-server# git checkout fix-export-import
    -bash: root@ubuntu:/opt/xo-server#: File o directory non esistente
    


  • @olivierlambert

    sorry... (copy and paste error...)
    I redo all but from the "git checkout" command I receive a warning...

    root@ubuntu:/opt/xo-server# git pull
    remote: Counting objects: 150, done.
    remote: Compressing objects: 100% (58/58), done.
    remote: Total 150 (delta 101), reused 127 (delta 92), pack-reused 0
    Ricezione degli oggetti: 100% (150/150), 119.97 KiB | 0 bytes/s, done.
    Risoluzione dei delta: 100% (101/101), completed with 32 local objects.
    Da https://github.com/vatesfr/xo-server
       2d39a1a..dc9af66  backupRestoration -> origin/backupRestoration
       edbeb55..eac4178  cbs-exposeVmAppliance -> origin/cbs-exposeVmAppliance
     * [nuovo branch]    fix-export-import -> origin/fix-export-import
     * [nuovo branch]    fixCreateVm-resourceSet -> origin/fixCreateVm-resourceSet
       de7672b..ff16533  next-release -> origin/next-release
       f217c84..9ac1545  nr-xosan2  -> origin/nr-xosan2
    Already up-to-date.
    root@ubuntu:/opt/xo-server#
    
    root@ubuntu:/opt/xo-server# git branch -a
    * stable
      remotes/origin/HEAD -> origin/next-release
      remotes/origin/architecture-rework
      remotes/origin/babel-config
      remotes/origin/backup-cleanup
      remotes/origin/backupRestoration
      remotes/origin/cancelable-backups
      remotes/origin/cancellable-vm-import
      remotes/origin/cbs-exposeVmAppliance
      remotes/origin/collection-patch
      remotes/origin/disable-quiesce-fallback
      remotes/origin/extract-xo-fs-to-module
      remotes/origin/fix-export-import
      remotes/origin/fix-registerUser
      remotes/origin/fixCreateVm-resourceSet
      remotes/origin/fixResourceSet
      remotes/origin/force-ext4
      remotes/origin/jobTimezone
      remotes/origin/master
      remotes/origin/next-release
      remotes/origin/nr-try-fix-xosan-beta
      remotes/origin/nr-update-noVNC
      remotes/origin/nr-xosan
      remotes/origin/nr-xosan2
      remotes/origin/olt-detectUnheathlyChain
      remotes/origin/olt-testsrstats
      remotes/origin/patch
      remotes/origin/revert-CR-DR-job
      remotes/origin/stable
      remotes/origin/utils
      remotes/origin/why-is-node-running
    
    
    root@ubuntu:/opt/xo-server# git checkout fix-export-import
    error: Your local changes to the following files would be overwritten by checkout:
            package.json
    Please commit your changes or stash them before you switch branches.
    Aborting
    root@ubuntu:/opt/xo-server#


  • Well your git is not doing this automatically, that's why it's hard to support non-XOA installs ^^

    So do a:
    git checkout origin/fix-export-import



  • @olivierlambert

    same message:

    root@ubuntu:/opt/xo-server# git checkout origin/fix-export-import
    error: Your local changes to the following files would be overwritten by checkout:
            package.json
    Please commit your changes or stash them before you switch branches.
    Aborting
    root@ubuntu:/opt/xo-server#


  • You'll need to stash your changes before you can switch branches -- git stash -u


Log in to reply