Delta Backups Stuck at Merge



  • I am also having issues with delta backups stalling at the merge step when at the retention limit. I've actually been having this problem since July, which I believe coincides with switching to backup-ng. Delta backups were working perfectly under the old system since the feature was introduced.

    As part of my troubleshooting I setup a new backup job for a brand new vm, that had no existing snapshots, backups, etc. Still stuck at merge.

    Created a brand new XO vm running both Debian and Ubuntu LTS, build from the Jarli01 script...same problem. So I don't believe it is anything related to my xenserver (7.2) or xo vm. Tried dropping back to the node 6x release that xoa uses, but yarn fails during the build process. IIRC, some months ago this happened during xo upgrades and I had to move to the 8x LTS release to successfully build.

    XOA runs fine, but the trial period will expire soon. for my single server at home, running from the sources has been a great solution.

    [root@xo ~]# node -v
    v8.11.4
    [root@xo ~]# npm -v
    5.6.0
    [root@xo ~]# yarn -v
    1.9.4
    [root@xo ~]# n -V
    2.1.8
    


  • manually ran a job at 13:19 and set the job to run every hour. the 14:00 job ran to completion. the 15:00 job is still "Started" at 4:45pm. I have it set to retain two delta's. Currently I have two snapshots, with three .vhds and two jsons on my nfs mount.

    root@xoce:~# journalctl -f -u xo-server
    Sep 09 13:19:31 xoce xo-server[11392]: 2018-09-09T17:19:31.891Z xo:xapi Snapshotting VM hassio as [XO Backup test] hassio
    Sep 09 13:19:37 xoce xo-server[11392]: 2018-09-09T17:19:37.197Z xo:xapi exporting VDI hass_os
    Sep 09 14:00:00 xoce xo-server[11392]: 2018-09-09T18:00:00.034Z xo:xapi Snapshotting VM hassio as [XO Backup test] hassio
    Sep 09 14:00:05 xoce xo-server[11392]: 2018-09-09T18:00:05.457Z xo:xapi exporting VDI hass_os (from base hass_os)
    Sep 09 14:00:19 xoce xo-server[11392]: 2018-09-09T18:00:19.535Z xo:xapi Deleting VM [XO Backup test] hassio
    Sep 09 14:00:19 xoce xo-server[11392]: 2018-09-09T18:00:19.633Z xo:xapi Deleting VDI hass_os
    Sep 09 15:00:00 xoce xo-server[11392]: 2018-09-09T19:00:00.026Z xo:xapi Snapshotting VM hassio as [XO Backup test] hassio
    Sep 09 15:00:05 xoce xo-server[11392]: 2018-09-09T19:00:05.520Z xo:xapi exporting VDI hass_os (from base hass_os)
    


  • I've found a couple of threads on this now that seem to end with no resolution... has anybody got a solution to this? I'm having the same problem - delta backups fine until retention limit reached, where the process hangs at merge.

    xcp-ng 7.5
    Ubuntu 18.04.01 fully updated
    xo-server 5.25.1 from sources (jarlii script)
    NFS remote on Synology NAS

    I'm not sure that the merge is actually failing - it seems to be behaving more like the backup job is not reporting back to XO correctly. The new backup is available and the oldest one is removed from the restore list, but the other deltas all seem to be OK. When I click cancel, XO doesn't show any change in the status (still sitting as started) but after a reboot it correctly reports it as interrupted.

    I'm trying to figure out how to test this, particularly the question of whether the backup is run correctly or not. At this stage I think the main problem is that the while the process is reporting as hung in XO, it prevents future backups from happening; it doesn't seem like data is going missing or being corrupted.



  • @sccfit said in Delta Backups Stuck at Merge:

    xo-server 5.25.1

    You should update to the latest version and try again. 🙂



  • whoops... didn't realise there was a more recent version... comes from being a cheapskate not paying for support. Will try that shortly.



  • Thanks... update to 5.28.0 seems to have sorted it 🙂



  • @sccfit said in Delta Backups Stuck at Merge:

    whoops... didn't realise there was a more recent version... comes from being a cheapskate not paying for support. Will try that shortly.

    You can also set a cron job to update 🙂



  • Spoke too soon... have updated to 5.28.0, last nights backup has the VMs stuck at merge again 😞



  • updated to 5.28.0 as well with no improvement in merging the VDI's. XOA works fine so clearly the code is working. i've tried different distro's (centos, debian, ubuntu) and versions of node with no fix so i don't know what else to do. i get stuck with a VM with 2 snapshots (instead of one), three .json's (correct) and four VDI's (instead of the 3 i set my retention to).

    this is just my personal home server so not a HUGE deal if delta's don't work, but it WAS working since the feature was added and even when backup-ng was introduced.

    does anyone actually have delta's working with a source build? if so, what versions of node, npm, yarn?



  • Yes, it works fine for me. What version of libvhdi-utils are you running?



  • under Centos 7 I don't have that installed. it isn't even a standard package, even in epel. is that a dependency? if so, that must be recent as this used to work without it. can't check my test vm's of either debian or ubuntu as i destroyed them, but they should have had it installed.



  • I believe that is where vhdimount comes from. Can you check if this tool is installed and the version?



  • i installed libvhdi and libvhdi-tools, both 20170223, in my centos 7 vm so i now have vhdimount and vhdiinfo. blew away all snaphots and backups for the vm i'm trying to backup and it is still hanging at the merge step once the retention limit is met.

    i wonder if there is a way to turn up the verbosity on what xo is actually trying to do? for example does it even call vhdimount? i've never seen anything like that in the logs, even with "verboseApiLogsOnErrors: true" in my config.



  • built a brand new ubuntu 18.10 server vm and installed xo 5.28 via the jarli01 script. delta's still hang at the merge step.

    root@xoce:~# node -v
    v8.12.0
    root@xoce:~# npm -v
    6.4.1
    root@xoce:~# yarn -v
    1.10.1
    root@xoce:~# n -V
    2.1.12
    root@xoce:~# dpkg -l | grep vhdi
    ii  libvhdi-utils                         20170223-3                        amd64        Virtual Hard Disk image format access library -- Utilities
    ii  libvhdi1:amd64                        20170223-3                        amd64        Virtual Hard Disk image format access library
    


  • Here are mine --

    root@ubuntu:/# node -v
    v8.11.4
    root@ubuntu:/# npm -v
    5.6.0
    root@ubuntu:/# yarn -v
    1.10.1
    root@ubuntu:/# n -V
    2.1.8
    root@ubuntu:/# dpkg -l | grep vhdi
    ii  libvhdi-utils                              20170223-3                                   amd64        Virtual Hard Disk image format access library -- Utilities
    ii  libvhdi1:amd64                             20170223-3                                   amd64        Virtual Hard Disk image format access library
    


  • You could also try this to get more detailed debugging --

    • Edit the file packages/vhd-lib/src/vhd.js with your favorite editor
    • Change the line -- const VHD_UTIL_DEBUG = 0 to const VHD_UTIL_DEBUG = 1
    • Rebuild XO with yarn; yarn build
    • Restart XO
    • Run another backup


  • i am at a loss as to why this is working for you but not me or the handful of folks who have chimed in here.

    again, a brand new ubuntu 18.04 vm with xo built from sources refuses to merge the vhd once it hits the retention limit. i rebuilt with your tip to add debugging to the log but nothing there looks helpful. it appears to never even TRY to merge the 3rd vhd and leave only 2 (my retention limit test). i see some vhd-merge debug language, but it is identical between the 2nd delta backup job and 3rd, but i am expecting to see some additional logging for the actual merge and it just isn't there.

    Oct 28 12:14:51 xoce xo-server[3775]: 2018-10-28T12:14:51.724Z - xo:xapi - [DEBUG] Snapshotting VM hassio as [XO Backup test] hassio
    Oct 28 12:14:57 xoce xo-server[3775]: 2018-10-28T12:14:57.092Z - xo:xapi - [DEBUG] exporting VDI hass_os
    Oct 28 12:19:12 xoce xo-server[3775]: [vhd-merge]End of headers: 22528.
    Oct 28 12:19:12 xoce xo-server[3775]: [vhd-merge]End of data: 16052090.
    Oct 28 12:19:12 xoce xo-server[3775]: [vhd-merge]Write footer at: 8218670080 (checksum=4294963230). (data=636f6e6563746978000000020001000000000000000002000000000063616d6c00000001000000000000000280000000000000028
    Oct 28 12:19:12 xoce xo-server[3775]: [vhd-merge]_write offset=0 size=512
    Oct 28 12:19:12 xoce xo-server[3775]: [vhd-merge]_write offset=8218670080 size=512
    Oct 28 13:28:24 xoce xo-server[3775]: 2018-10-28T13:28:24.256Z - xo:xapi - [DEBUG] Snapshotting VM hassio as [XO Backup test] hassio
    Oct 28 13:28:29 xoce xo-server[3775]: 2018-10-28T13:28:29.715Z - xo:xapi - [DEBUG] exporting VDI hass_os (from base hass_os)
    Oct 28 13:28:58 xoce xo-server[3775]: [vhd-merge]_write offset=1536 size=40
    Oct 28 13:28:58 xoce xo-server[3775]: [vhd-merge]Write header at: 512 (checksum=4294960105). (data=6378737061727365ffffffffffffffff0000000000000800000100000000140000200000ffffe3e9f01da8af6b994f6985886faaefcd846f
    Oct 28 13:28:58 xoce xo-server[3775]: [vhd-merge]_write offset=512 size=1024
    Oct 28 13:28:58 xoce xo-server[3775]: [vhd-merge]End of headers: 263680.
    Oct 28 13:28:58 xoce xo-server[3775]: [vhd-merge]End of data: 1339763.
    Oct 28 13:28:58 xoce xo-server[3775]: [vhd-merge]Write footer at: 685958656 (checksum=4294963390). (data=636f6e6563746978000000020001000000000000000002000000000063616d6c000000010000000000000002800000000000000280
    Oct 28 13:28:58 xoce xo-server[3775]: [vhd-merge]_write offset=0 size=512
    Oct 28 13:28:58 xoce xo-server[3775]: [vhd-merge]_write offset=685958656 size=512
    Oct 28 13:28:58 xoce xo-server[3775]: [vhd-merge]End of headers: 263680.
    Oct 28 13:28:58 xoce xo-server[3775]: [vhd-merge]End of data: 1339763.
    Oct 28 13:28:58 xoce xo-server[3775]: [vhd-merge]Write footer at: 685958656 (checksum=4294963635). (data=636f6e6563746978000000020001000000000000000002000000000063616d6c000000010000000000000002800000000000000280
    Oct 28 13:28:58 xoce xo-server[3775]: [vhd-merge]_write offset=0 size=512
    Oct 28 13:28:58 xoce xo-server[3775]: [vhd-merge]_write offset=685958656 size=512
    Oct 28 13:28:58 xoce xo-server[3775]: 2018-10-28T13:28:58.812Z - xo:xapi - [DEBUG] Deleting VM [XO Backup test] hassio
    Oct 28 13:28:58 xoce xo-server[3775]: 2018-10-28T13:28:58.891Z - xo:xapi - [DEBUG] Deleting VDI hass_os
    Oct 28 13:58:20 xoce xo-server[3775]: 2018-10-28T13:58:20.778Z - xo:xapi - [DEBUG] Snapshotting VM hassio as [XO Backup test] hassio
    Oct 28 13:58:27 xoce xo-server[3775]: 2018-10-28T13:58:27.777Z - xo:xapi - [DEBUG] exporting VDI hass_os (from base hass_os)
    Oct 28 13:58:47 xoce xo-server[3775]: [vhd-merge]_write offset=1536 size=40
    Oct 28 13:58:47 xoce xo-server[3775]: [vhd-merge]Write header at: 512 (checksum=4294960493). (data=6378737061727365ffffffffffffffff0000000000000800000100000000140000200000ffffe56d390a627fea4345a48c77e87c12b7db1a
    Oct 28 13:58:47 xoce xo-server[3775]: [vhd-merge]_write offset=512 size=1024
    Oct 28 13:58:47 xoce xo-server[3775]: [vhd-merge]End of headers: 263680.
    Oct 28 13:58:47 xoce xo-server[3775]: [vhd-merge]End of data: 778474.
    Oct 28 13:58:47 xoce xo-server[3775]: [vhd-merge]Write footer at: 398578688 (checksum=4294963134). (data=636f6e6563746978000000020001000000000000000002000000000063616d6c000000010000000000000002800000000000000280
    Oct 28 13:58:47 xoce xo-server[3775]: [vhd-merge]_write offset=0 size=512
    Oct 28 13:58:47 xoce xo-server[3775]: [vhd-merge]_write offset=398578688 size=512
    Oct 28 13:58:47 xoce xo-server[3775]: [vhd-merge]End of headers: 263680.
    Oct 28 13:58:47 xoce xo-server[3775]: [vhd-merge]End of data: 778474.
    Oct 28 13:58:47 xoce xo-server[3775]: [vhd-merge]Write footer at: 398578688 (checksum=4294963485). (data=636f6e6563746978000000020001000000000000000002000000000063616d6c000000010000000000000002800000000000000280
    Oct 28 13:58:47 xoce xo-server[3775]: [vhd-merge]_write offset=0 size=512
    Oct 28 13:58:47 xoce xo-server[3775]: [vhd-merge]_write offset=398578688 size=512
    


  • Can you show us the backup job definition? What is your backup retention value? Also, have you changed the Concurrency value?



  • nothing special as it is one i created just as a test. delta backup to an nfs share for a single vm, retention set at 2. concurrency at the default



  • I just built a brand new VM from scratch and the merge still worked flawlessly. 🤷🏻♂

    Are you still running XS 7.2? If so, do you have the ability to test with a newer version of XS / XCP?


Log in to reply