Delta Backups Stuck at Merge



  • @olivierlambert I have installed the xen-orchestra with Jarli01's install script from github when the xo-server version was 5.23 then updated that to 5.24.2 and currently to 5.25.2 today. Also tried with different Xen server versions and XCP-NG latest as well with fresh orchestra installations. Before that tried with installation from source as well with same results. Delta backup seems to work fine until it hits retention number which I have set to 2 for testing.



  • @Rachendra can you just do quickly a test with a XOA (register for the free trial, it's free and you'll have all features). If you can reproduce, then we'll be sure about this.



  • @olivierlambert I will try out the free trial and let you know.



  • Thanks!



  • @olivierlambert No, not currently experiencing issues w/ merging on delta backups.

    @Rachendra What base OS did you use when you created your XO VM?



  • @Danp using ubuntu 16.04 as base os for xo vm.



  • @rachendra Suggest that you switch to a more recent version. I'm using 18.04 and not seeing these issues.



  • @Danp upgraded ubuntu 16.04 to 18.04 still having same issue. So, building new one from scratch. Can you share with me the nodejs and yarn version you used to compile the XO in 18.04? i am suspecting the problem with nodejs. @olivierlambert used trial XOA. Didn't have the issue. Now trying to build new one with nodejs v6 as used in XOA.



  • Thanks for the test @Rachendra so it's likely a problem with your setup and/or the sources directly (but if it was a "source" problem, the problem would occur for @Danp too).

    Now you know why we only make pro support for XOA installs: the environment can lead to side effects that are very hard to debug… With XOA, everything is in a known env, which facilitate our short fix time.



  • @rachendra Technically, I'm running Ubuntu 18.04.1. Here's the version info you asked for

    $ node -v
    v8.11.4
    $ yarn -v
    1.9.4
    

    Have you tried installing XO with one of the existing install scripts, such as this one?



  • I am also having issues with delta backups stalling at the merge step when at the retention limit. I've actually been having this problem since July, which I believe coincides with switching to backup-ng. Delta backups were working perfectly under the old system since the feature was introduced.

    As part of my troubleshooting I setup a new backup job for a brand new vm, that had no existing snapshots, backups, etc. Still stuck at merge.

    Created a brand new XO vm running both Debian and Ubuntu LTS, build from the Jarli01 script...same problem. So I don't believe it is anything related to my xenserver (7.2) or xo vm. Tried dropping back to the node 6x release that xoa uses, but yarn fails during the build process. IIRC, some months ago this happened during xo upgrades and I had to move to the 8x LTS release to successfully build.

    XOA runs fine, but the trial period will expire soon. for my single server at home, running from the sources has been a great solution.

    [root@xo ~]# node -v
    v8.11.4
    [root@xo ~]# npm -v
    5.6.0
    [root@xo ~]# yarn -v
    1.9.4
    [root@xo ~]# n -V
    2.1.8
    


  • manually ran a job at 13:19 and set the job to run every hour. the 14:00 job ran to completion. the 15:00 job is still "Started" at 4:45pm. I have it set to retain two delta's. Currently I have two snapshots, with three .vhds and two jsons on my nfs mount.

    root@xoce:~# journalctl -f -u xo-server
    Sep 09 13:19:31 xoce xo-server[11392]: 2018-09-09T17:19:31.891Z xo:xapi Snapshotting VM hassio as [XO Backup test] hassio
    Sep 09 13:19:37 xoce xo-server[11392]: 2018-09-09T17:19:37.197Z xo:xapi exporting VDI hass_os
    Sep 09 14:00:00 xoce xo-server[11392]: 2018-09-09T18:00:00.034Z xo:xapi Snapshotting VM hassio as [XO Backup test] hassio
    Sep 09 14:00:05 xoce xo-server[11392]: 2018-09-09T18:00:05.457Z xo:xapi exporting VDI hass_os (from base hass_os)
    Sep 09 14:00:19 xoce xo-server[11392]: 2018-09-09T18:00:19.535Z xo:xapi Deleting VM [XO Backup test] hassio
    Sep 09 14:00:19 xoce xo-server[11392]: 2018-09-09T18:00:19.633Z xo:xapi Deleting VDI hass_os
    Sep 09 15:00:00 xoce xo-server[11392]: 2018-09-09T19:00:00.026Z xo:xapi Snapshotting VM hassio as [XO Backup test] hassio
    Sep 09 15:00:05 xoce xo-server[11392]: 2018-09-09T19:00:05.520Z xo:xapi exporting VDI hass_os (from base hass_os)
    


  • I've found a couple of threads on this now that seem to end with no resolution... has anybody got a solution to this? I'm having the same problem - delta backups fine until retention limit reached, where the process hangs at merge.

    xcp-ng 7.5
    Ubuntu 18.04.01 fully updated
    xo-server 5.25.1 from sources (jarlii script)
    NFS remote on Synology NAS

    I'm not sure that the merge is actually failing - it seems to be behaving more like the backup job is not reporting back to XO correctly. The new backup is available and the oldest one is removed from the restore list, but the other deltas all seem to be OK. When I click cancel, XO doesn't show any change in the status (still sitting as started) but after a reboot it correctly reports it as interrupted.

    I'm trying to figure out how to test this, particularly the question of whether the backup is run correctly or not. At this stage I think the main problem is that the while the process is reporting as hung in XO, it prevents future backups from happening; it doesn't seem like data is going missing or being corrupted.



  • @sccfit said in Delta Backups Stuck at Merge:

    xo-server 5.25.1

    You should update to the latest version and try again. πŸ™‚



  • whoops... didn't realise there was a more recent version... comes from being a cheapskate not paying for support. Will try that shortly.



  • Thanks... update to 5.28.0 seems to have sorted it πŸ™‚



  • @sccfit said in Delta Backups Stuck at Merge:

    whoops... didn't realise there was a more recent version... comes from being a cheapskate not paying for support. Will try that shortly.

    You can also set a cron job to update πŸ™‚



  • Spoke too soon... have updated to 5.28.0, last nights backup has the VMs stuck at merge again 😞



  • updated to 5.28.0 as well with no improvement in merging the VDI's. XOA works fine so clearly the code is working. i've tried different distro's (centos, debian, ubuntu) and versions of node with no fix so i don't know what else to do. i get stuck with a VM with 2 snapshots (instead of one), three .json's (correct) and four VDI's (instead of the 3 i set my retention to).

    this is just my personal home server so not a HUGE deal if delta's don't work, but it WAS working since the feature was added and even when backup-ng was introduced.

    does anyone actually have delta's working with a source build? if so, what versions of node, npm, yarn?



  • Yes, it works fine for me. What version of libvhdi-utils are you running?


Log in to reply