Continuous Replication stopped working



  • Dear Community,

    we recently upgraded a 3 node Xenserver pool from XS71ECU LTSR CU1 to XS71ECU2 incl. follow-up patch XS71ECU2001. We are still using xo-server/web 5.21.0 with this pool.

    Our Continuous Replication jobs stopped working after the upgrade. It looks like the whole export/import works, the jobs is displayed until completed under tasks but the backup job itself always fails with:

    VDI_IO_ERROR(Device I/O errors)

    I've looked at all the logs and the only thing I found is in orchestra.log, no hints in SMlog and xensource on the hosts itself. (I noticed the timestamps in orchestra.log are UTC, instead of CET, is this supposed or is the timezone not read correctly?)

    The following output occured right after the CR task in XO was finished and cleared from tasks.

    2019-03-28T14:53:27.175Z xo:xapi Deleting VDI FR2SGW02
    2019-03-28T14:53:28.326Z xo:xapi Deleting VM [XO Backup CR_daily_test_fr2syn03] FR2SGW02
    2019-03-28T14:53:28.556Z xo:xapi Deleting VM [XO Backup CR_daily_test_fr2syn03] FR2SGW02
    2019-03-28T14:53:28.747Z xo:xapi Deleting VDI FR2SGW02
    backup report error: TypeError: Cannot read property 'calls' of undefined
        at BackupReportsXoPlugin._listener (/usr/local/lib/node_modules/xo-server-backup-reports/src/index.js:69:19)
        at /usr/local/lib/node_modules/xo-server-backup-reports/src/index.js:57:47
        at new Promise (/opt/xen-orchestra/node_modules/core-js/modules/es6.promise.js:177:7)
        at BackupReportsXoPlugin._wrapper (/usr/local/lib/node_modules/xo-server-backup-reports/src/index.js:57:11)
        at emitMany (events.js:156:13)
        at Xo.emit (events.js:230:7)
        at Xo.patchedError [as emit] (/opt/xen-orchestra/packages/xo-server/bin/xo-server:27:17)
        at /opt/xen-orchestra/packages/xo-server/src/xo-mixins/jobs/index.js:289:10
        at Generator.next (<anonymous>)
        at step (/opt/xen-orchestra/packages/xo-server/dist/xo-mixins/jobs/index.js:32:221)
        at _next (/opt/xen-orchestra/packages/xo-server/dist/xo-mixins/jobs/index.js:32:409)
        at run (/opt/xen-orchestra/node_modules/core-js/modules/es6.promise.js:75:22)
        at /opt/xen-orchestra/node_modules/core-js/modules/es6.promise.js:92:30
        at flush (/opt/xen-orchestra/node_modules/core-js/modules/_microtask.js:18:9)
        at _combinedTickCallback (internal/process/next_tick.js:131:7)
        at process._tickCallback (internal/process/next_tick.js:180:9)
    

    I tried to delete older CR copies from the SR, switched to another NFS SR as well but the problem persists.
    Our other backupjobs (snapshot, delta) are working properly, so I just guess there may be a compatibility problem between latest Xenserver 7.1 CU2 and our rather old xo-server/web (5.21.0 from sources).

    I am lost now and have no clue where to look further into it.

    We decided to delay the upgrades on our other pools until we found a solution.

    Thanks for any hints how to solve this.

    Regards,
    -Toni



  • @tts said in Continuous Replication stopped working:

    We are still using xo-server/web 5.21.0 with this pool.

    Why? That's like 9 months behind on updates!

    There was a similar reported issue back that was fixed several months ago.
    https://github.com/vatesfr/xen-orchestra/issues/3862

    Time to take a snapshot of your XO VM and then upgrade.



  • I thought about that, thank you for clarifying this for me.

    Why? That's like 9 months behind on updates!

    I'm yet new to these systems and still learning the concepts and tools after I took control of these systems recently.

    I updated xo-server/web on our local installation (v5.32.2) to test things first.
    But now I am unsure how to update the plugins seperately.

    I've ran npm i xo-server-backup-reports inside packages/xo-server but it looks like it installs the same (old?) version 0.7.0 of the plugin.

    I did not find hints in the documentation as it looks that "install from sources" section only covers the installation of xo-server/web, but not the process of updating plugins.

    I've read something about plugins not maintained by xo-team, so how to update them correctly?

    The plugins are located at
    xen-orchestra/packages/xo-server/node_modules/...
    and
    /usr/local/lib/node_modules/...

    There was some updated documentation mentioned by Olivier on the forum, but was linking to the older repository and date back to 2015.

    Updated the doc accordingly: https://github.com/vatesfr/xo-server-backup-reports/blob/master/README.md

    I am very thankful for some hints how to update them correctly.

    kind regards,
    -Toni



  • @tts said in Continuous Replication stopped working:

    I updated xo-server/web on our local installation (v5.32.2) to test things first.

    What process / steps did you use to do this?

    But now I am unsure how to update the plugins seperately.
    I've ran npm i xo-server-backup-reports inside packages/xo-server but it looks like it installs the same (old?) version 0.7.0 of the plugin.

    None of this is needed because the plugin source should already be updated from your steps above.

    I've read something about plugins not maintained by xo-team

    Not true.... unless you are referring to the version on NPM.

    The plugins are located at
    xen-orchestra/packages/xo-server/node_modules/...
    and
    /usr/local/lib/node_modules/...

    The first directory is the source code that came from Github. The second is one of the locations (I believe there are two) where XO looks for plugins to load.

    Instead of copying the source into this directory, it's better to create a symlink so that the desired plugins are loaded. An example would be --

    ln -s /opt/xen-orchestra/packages/xo-server-usage-report /usr/local/lib/node_modules/

    HTH,

    Dan



  • This helps, maybe just small steps, but it helps, thank you 😉

    What process / steps did you use to do this?

    inside xen-orchestra folder:

    git pull --ff-only
    yarn
    yarn build
    

    None of this is needed

    Looks like I accidently mixed yarn and npm and ended up overwriting the current version with the 2years old npm-hosted packages, I did read a lot of quite old posts, maybe I mixed the wrong stuff together - time to revert the snapshot.

    I noticed the package under /opt/xen-orchestra/packages/xo-server-backup-reports was indeed at 0.15 but there was another package located under /opt/xen-orchestra/packages/xo-server/node_modules/.. which was still at 0.5

    -Toni



  • Dear Community,

    we have now updated our XO from sources to (xo-server 5.39.0,xo-web 5.39.1) but we are still unable to get CR on a nfs share working. Xen server is 7.1 CU2.

    We hooked up an older xen server, which has plenty of local lvm storage available and used this as the CR target and everything works as expected.

    If we switch to the nfs based storage the CR job finishes but is instantly deleting the imported VM/VDI afterwards. The backup job then states: VDI_IO_ERROR(Device I/O errors)

    I am not seeing any erros in orchestra.log, the snapshot creation and deletion is logged as followed:

    2019-04-29T12:26:36.683Z - xo:xapi - [DEBUG] Snapshotting VM HALOpenVAS01 as [XO Backup CR_TEST_NAS02_NFS3] HALOpenVAS01
    2019-04-29T12:26:38.114Z - xo:xapi - [DEBUG] Creating VM [XO Backup CR_TEST_NAS02_NFS3] HALOpenVAS01
    2019-04-29T12:26:38.322Z - xo:xapi - [DEBUG] Creating VDI HALOpenVAS01_01 on XENCR_NAS02
    2019-04-29T12:26:38.518Z - xo:xapi - [DEBUG] Creating VBD for VDI HALOpenVAS01_01 on VM [Importing…] HALOpenVAS01 - CR_TEST_NAS02_NFS3 - (20190429T122638Z)
    2019-04-29T12:26:38.536Z - xo:xapi - [DEBUG] exporting VDI HALOpenVAS01_01
    2019-04-29T12:26:38.537Z - xo:xapi - [DEBUG] Creating VIF for VM [XO Backup CR_TEST_NAS02_NFS3] HALOpenVAS01 on network SUPPORT
    
    root@192.168.x.x Xapi#putResource /import_raw_vdi/ missing length
    
    2019-04-29T12:30:11.510Z - xo:xapi - [DEBUG] Deleting VDI OpaqueRef:28e305f4-7491-b6b1-10e3-2db4eeb68b24
    2019-04-29T12:30:11.942Z - xo:xapi - [DEBUG] Deleting VM [Importing…] HALOpenVAS01 - CR_TEST_NAS02_NFS3 - (20190429T122638Z)
    2019-04-29T12:30:12.158Z - xo:xapi - [DEBUG] Deleting VM [XO Backup CR_TEST_NAS02_NFS3] HALOpenVAS01
    2019-04-29T12:30:12.209Z - xo:xapi - [DEBUG] Deleting VDI OpaqueRef:b085d833-d2f8-e5a8-92a1-a04ce9fc977b
    

    The only suspicious here is the root@ line with the missing length message.

    We are using a second nfs share for Deltabackups from the same storage device as the CR share and the backups are working.

    Currently I have no idea where to look further into this issue, any help is appreciated.

    Thanks
    -Toni



  • Hi,

    Citrix added something different in CU2, and we found a way to workaround it. If you have XOA with Pro Support, please open a support ticket so we can enable you the workaround. It's not deployed everywhere to be sure there isn't no edge cases with it, so we unlock it on "per ticket" basis 🙂

    Note: if you don't have support, you can modify yourself the xo-server config file and enable guessVhdSizeOnImport. But if it's for production usage, please consider to get pro support at least for your backups!



  • Thanks Olivier for the hint with the server config, setting the option makes our nfs based Continuous Replication working again.

    We already discussed a possible Pro support internally but if I understand the license model correctly, we have to go for the 6k a year option to be able to use Deltabackups and Continuous Replication with pro support enabled?

    This would mean, paying for it will reduce our backup functionality dramatically unless we choose the most expensive option?



  • That's correct:

    • Delta is in Enterprise
    • CR is in Premium

    This also bring: turnkey XOA, web updater, QA and support (and also more features and services coming, like backup proxies, app catalog etc.)

    edit: and our pricing is flat, regardless number of host. If you can't afford Premium, please contact us in private to discuss a possible discount 🙂


Log in to reply