Delta Backups Stuck at Merge



  • Using BackupNG the delta backups run flawlessly until they hit the retention number.

    The logs show the job as started and it is on the merge phase.

    There is no error message. Nothing is logged and the job stays like this in perpetuity. I have left the job for 48 hours just in case the merge was REALLY slow.

    I am storing data locally. I have even wiped out my install, cleared data directory and wiped redis. Started a new instance and had the exact same issue.

    Compiled from source
    Ubuntu 17.10
    Xo Web 5.20
    Xo Server 5.20
    Xensrver 7.5 unlicensed (free?)

    I searched the forum but since no error is produced I'm stuck. Any help would be appreciated.



  • Just updated to 5.21 (web and server). Still had the same result

    Transfer is successful but merge does not start or error out.



  • I would suggest kicking off a backup job and then start an SSH sesssion where you can monitor the XO logs.

    I use this command to monitor the logs --
    journalctl -u xo-server -f -n 50



  • Here is the logs from the last backup I started

    Jun 28 12:29:17 vpn xo-server[16743]: yarn run v1.7.0
    Jun 28 12:29:17 vpn xo-server[16743]: $ node bin/xo-server
    Jun 28 12:29:22 vpn xo-server[16743]: Thu, 28 Jun 2018 16:29:22 GMT app-conf /opt/xen-orchestra/packages/xo-server/config.json
    Jun 28 12:29:22 vpn xo-server[16743]: Thu, 28 Jun 2018 16:29:22 GMT app-conf /etc/xo-server/config.yaml
    Jun 28 12:29:22 vpn xo-server[16743]: 2018-06-28T16:29:22.775Z xo:main Configuration loaded.
    Jun 28 12:29:22 vpn xo-server[16743]: 2018-06-28T16:29:22.780Z xo:main Web server listening on http://[::]:80
    Jun 28 12:29:22 vpn xo-server[16743]: 2018-06-28T16:29:22.781Z xo:main Group changed to root
    Jun 28 12:29:22 vpn xo-server[16743]: 2018-06-28T16:29:22.781Z xo:main User changed to root
    Jun 28 12:29:22 vpn xo-server[16743]: 2018-06-28T16:29:22.842Z xo:hooks start start…
    Jun 28 12:29:22 vpn xo-server[16743]: 2018-06-28T16:29:22.964Z xo:hooks start finished
    Jun 28 12:29:22 vpn xo-server[16743]: 2018-06-28T16:29:22.965Z xo:hooks clean start…
    Jun 28 12:29:23 vpn xo-server[16743]: 2018-06-28T16:29:23.078Z xo:hooks clean finished
    Jun 28 12:29:23 vpn xo-server[16743]: 2018-06-28T16:29:23.155Z xo:main Setting up / → ../xo-web/dist/
    Jun 28 12:29:55 vpn xo-server[16743]: 2018-06-28T16:29:55.811Z xo:main + WebSocket connection (::ffff:10.8.0.6)
    Jun 28 12:30:28 vpn xo-server[16743]: 2018-06-28T16:30:28.940Z xo:xapi Snapshotting VM PBX as [XO Backup Test] PBX
    Jun 28 12:30:31 vpn xo-server[16743]: 2018-06-28T16:30:31.844Z xo:xapi exporting VDI PBXData (from base PBXData)
    Jun 28 12:30:31 vpn xo-server[16743]: 2018-06-28T16:30:31.846Z xo:xapi exporting VDI PBXBoot (from base PBXBoot)
    Jun 28 12:35:13 vpn xo-server[16743]: 2018-06-28T16:35:13.517Z xo:main - WebSocket connection (::ffff:10.8.0.6)



  • What kind of remote are you using?



  • @olivierlambert I have tried both NFS and local disk. Both have the same behavior. Is there a way to increase log level to make the steps more verbose?



  • Anything is Settings/logs?



  • There is nothing in Settings / Logs



  • Any update to this issue? As i am also having the same issue while running the community version of XO. Currently running latest xo-server 5.25.2 and xo-web 5.25.0 in ubuntu 16.04. As @mmartinwechu said, there is nothing in log. also journalctl generated below log:

    Aug 28 13:28:16 Orchestra xo-server[20104]: 2018-08-28T07:43:16.742Z xo:xapi Snapshotting VM MAIL9 as [XO Backup Delta Test] MAIL9
    Aug 28 13:28:21 Orchestra xo-server[20104]: 2018-08-28T07:43:21.213Z xo:xapi exporting VDI MAIL9 0 (from base MAIL9 0)
    Aug 28 13:29:14 Orchestra xo-server[20104]: 2018-08-28T07:44:14.931Z xo:perf blocked for 509ms

    Also attaching screenshot of the xo-web interface. 0_1535474505179_Capture.PNG
    0_1535474543700_log.PNG



  • I don't think we have any issue reported from XOA customers 😕

    @Danp do you experience problems on your side?



  • @olivierlambert I have installed the xen-orchestra with Jarli01's install script from github when the xo-server version was 5.23 then updated that to 5.24.2 and currently to 5.25.2 today. Also tried with different Xen server versions and XCP-NG latest as well with fresh orchestra installations. Before that tried with installation from source as well with same results. Delta backup seems to work fine until it hits retention number which I have set to 2 for testing.



  • @Rachendra can you just do quickly a test with a XOA (register for the free trial, it's free and you'll have all features). If you can reproduce, then we'll be sure about this.



  • @olivierlambert I will try out the free trial and let you know.



  • Thanks!



  • @olivierlambert No, not currently experiencing issues w/ merging on delta backups.

    @Rachendra What base OS did you use when you created your XO VM?



  • @Danp using ubuntu 16.04 as base os for xo vm.



  • @rachendra Suggest that you switch to a more recent version. I'm using 18.04 and not seeing these issues.



  • @Danp upgraded ubuntu 16.04 to 18.04 still having same issue. So, building new one from scratch. Can you share with me the nodejs and yarn version you used to compile the XO in 18.04? i am suspecting the problem with nodejs. @olivierlambert used trial XOA. Didn't have the issue. Now trying to build new one with nodejs v6 as used in XOA.



  • Thanks for the test @Rachendra so it's likely a problem with your setup and/or the sources directly (but if it was a "source" problem, the problem would occur for @Danp too).

    Now you know why we only make pro support for XOA installs: the environment can lead to side effects that are very hard to debug… With XOA, everything is in a known env, which facilitate our short fix time.



  • @rachendra Technically, I'm running Ubuntu 18.04.1. Here's the version info you asked for

    $ node -v
    v8.11.4
    $ yarn -v
    1.9.4
    

    Have you tried installing XO with one of the existing install scripts, such as this one?



  • I am also having issues with delta backups stalling at the merge step when at the retention limit. I've actually been having this problem since July, which I believe coincides with switching to backup-ng. Delta backups were working perfectly under the old system since the feature was introduced.

    As part of my troubleshooting I setup a new backup job for a brand new vm, that had no existing snapshots, backups, etc. Still stuck at merge.

    Created a brand new XO vm running both Debian and Ubuntu LTS, build from the Jarli01 script...same problem. So I don't believe it is anything related to my xenserver (7.2) or xo vm. Tried dropping back to the node 6x release that xoa uses, but yarn fails during the build process. IIRC, some months ago this happened during xo upgrades and I had to move to the 8x LTS release to successfully build.

    XOA runs fine, but the trial period will expire soon. for my single server at home, running from the sources has been a great solution.

    [root@xo ~]# node -v
    v8.11.4
    [root@xo ~]# npm -v
    5.6.0
    [root@xo ~]# yarn -v
    1.9.4
    [root@xo ~]# n -V
    2.1.8
    

Log in to reply