Delta Backups Stuck at Merge



  • yup, still 7.2. upgrading has been on my to-do list but haven't found the time yet. the fact that you have a new vm built and it is working i guess points to my XS install as a culprit, but i wouldn't have guessed that as it was working flawlessly until ~July, when the merges starting failing. not sure what, if anything, changed that could have affected this.



  • FYI, those merge are done on XO side, not XenServer.



  • @olivierlambert So what would you recommend to troubleshoot this further? Is there a particular module / library involved? Or could it be an rights issue with the NFS share?



  • Frankly, I have no idea (I'm following the topic from the start trying to see if there is something obvious there and it's a big "nope"). And I'm glad we offer only support on a known environment, because otherwise our support would have be drowned in the rabbit hole.



  • I think this is progress... I've managed to get the merge to fail with an error rather than hanging. Error presented in GUI is:

    ENOENT: no such file or directory, stat '/run/xo-server/mounts/e81e3a31-a4d8-4c33-a4f7-80ee74544ccb/xo-vm-backups/307118a8-4556-ab65-103d-29e857eb06a9/vdis/b7cf7a70-a640-4642-9204-c5d235a2dbc3/30ef9b86-c0b8-4704-a06c-afe03551f220/20181025T070023Z.vhd
    

    Does that indicate a permissions error? Journalctl -u xo-server is only showing a regular xo-server-cloud: fail to connect to updater; I assume this is standard for XO sources installs.

    To get the error, I've re-arranged my network topology (was working with 3 subnets, now only 2) and rebuilt VM from scratch in Ubuntu Server 18.04.1 using Jarlii01 script.



  • This is not related, probably a bug that have been on master for a day or so. Please update.



  • Thanks @olivierlambert, I've just run the backup job again and it is reporting a successful completion.

    One curiosity, which is possibly by design: while I have been attempting to resolve the merge issue, I extended the retention number out to 75, now that it is working, I've shortened it back up to 50, but there is still 75 backups available.

    I assume that it is keeping the 25 oldest ones and then creating a rolling snapshot of the last 50? I would have expected that setting the retention number to 50 would have removed all snapshots prior to the 50 most recent... for me it is not an issue, but can I delete the oldest ones without breaking the current chain?



  • This is a question for @julien-f



  • Just to tie this one off from my side...having reviewed the original VM, I had mistakenly installed onto a Ubuntu desktop VM, rather than a Ubuntu Server one. A new Ubuntu Server 18.04.1 install and the Jarli01 script appears to have resolved the issue with the merge; an update overnight has fixed the remaining errors.

    I'll reconfigure my network back how it was (probably tomorrow) to see if the topology creates issues, and report back after that.

    With respect to the issue of decreasing the retention number, everything seems to be working as expected now also; the correct number of backups are being held and they all appear to be sequential. Thanks to @olivierlambert @Danp @julien-f.



  • Same issue here.

    I'm using XenOrchestra since this May on Ubuntu 18.04 LTS (at least since 5.19). Installed (and updated) by the using the Script from Jarli01.
    Merge never worked for me!

    Last week i decided rebuild my XO Server to (hopefully) resolve this. But i had no success.

    1. Ubuntu Server 18.04 - no updates - Jarli01 Script -> merge does not work
    2. OS Update installed -> merge does not work

    I discovered that @sccfit was able to resolve it by using Ubuntu 18.04.1 -> I downloaded a fresh copy of Ubuntu Server 18.04.1

    1. Ubuntu Server 18.04.1 - no updates - Jarli01 Script -> merge does not work

    2. OS Updates installed -> merge does not work

    3. tried a different NFS target (on another NAS) -> merge does not work

    4. attached a local drive and used it as Backup target -> merge does not work

    5. Updated node.js with Jarli01 Script to "lts" -> XO does not work anymore (nodejs 10)

    6. Ubuntu Server 14.04 - no updates - Jarli01 Script -> merge does not work

    7. OS Updates installed -> merge does not work

    Still no success. I wonderd what the difference might be, in what im doing, setting the Server up. The only thing that came to my mind was the locale (my standard is de_CH).

    1. Ubuntu Server 18.04.1 with en_US locale - no updates - Jarli01 Script -> merge does not work
    2. OS Updates installed -> merge does not work
    3. Force rebuild with Jarli01 Script -> merge does not work

    Finally I decided to rebuild again, with my prefered standard Settings and for a change, to install the Updates in the beginning.

    1. Ubuntu Server 18.04.1 - OS Updates installed > reboot - Jarli01 Script -> merge does not work

    I don't really know what else to try anymore. Any ideas?



  • Also having this issue.
    Fully up to date 16.04.5 , 5.29.4 from sources (Jarli).
    Everything is fine until I hit retention (2).



  • I also saw Danp asking for versions so I'll drop that in too:

    xoaadmin@XOAFS:~$ node -v
    v8.12.0
    xoaadmin@XOAFS:~$ npm -v
    6.4.1
    xoaadmin@XOAFS:~$ yarn -v
    1.12.3
    xoaadmin@XOAFS:~$ n -V
    2.1.12
    xoaadmin@XOAFS:~$ dpkg -l | grep vhdi
    ii libvhdi-utils 20160108-1 amd64 Virtual Hard Disk image format access library -- Utilities
    ii libvhdi1:amd64 20160108-1 amd64 Virtual Hard Disk image format access library



  • @astrugatch said in Delta Backups Stuck at Merge:

    xoaadmin@XOAFS:~$ dpkg -l | grep vhdi
    ii libvhdi-utils 20160108-1 amd64 Virtual Hard Disk image format access library -- Utilities
    ii libvhdi1:amd64 20160108-1 amd64 Virtual Hard Disk image format access library

    I believe you need a newer version. Here's my details --

    dpkg -l | grep vhdi
    ii  libvhdi-utils                         20170223-3                        amd64        Virtual Hard Disk image format access library -- Utilities
    ii  libvhdi1:amd64                        20170223-3                        amd64        Virtual Hard Disk image format access library
    


  • @Danp

    The latest version in apt is 20160108-1 so this might be my issue. Are you building libvhdi-utils from source in the appliance or is there another source that I should be using?



  • Appears the newer version isn't available for Ubuntu 16.x -- https://packages.ubuntu.com/search?keywords=libvhdi-utils

    Time to move to a newer base OS. 🙂



  • @Danp

    Yeah, I noticed, that and decided to try Debian 9. Now I'm hitting the issue with the har-validator file not downloading correctly. Can't seem to win on this one.



  • @Danp
    What distro are you running? On Debian 9 I am still getting 20160424-1 from apt.



  • Ubuntu 18.10



  • here are my versions on Ubuntu 18.10.1

    user@xo:~$ node -v
    v8.12.0
    user@xo:~$ npm -v
    6.4.1
    user@xo:~$ yarn -v
    1.12.3
    user@xo:~$ n -V
    2.1.12
    user@xo:~$ dpkg -l | grep vhdi
    ii libvhdi-utils 20170223-3 amd64 Virtual Hard Disk image format access library -- Utilities
    ii libvhdi1:amd64 20170223-3 amd64 Virtual Hard Disk image format access library

    i just updated to xo-server 5.30.1 and xo-web 5.30.0
    but it still does not merge



  • Hi Everyone. Back after long time to see any update on this matter. I am still having same problem with Delta Backup even after countless setups of both XO, Xenserver and XCP-NG. Finally I opted for continuous replication. It works fine and have no issues with retention.


Log in to reply