Delta Backups Stuck at Merge
Using BackupNG the delta backups run flawlessly until they hit the retention number.
The logs show the job as started and it is on the merge phase.
There is no error message. Nothing is logged and the job stays like this in perpetuity. I have left the job for 48 hours just in case the merge was REALLY slow.
I am storing data locally. I have even wiped out my install, cleared data directory and wiped redis. Started a new instance and had the exact same issue.
Compiled from source
Xo Web 5.20
Xo Server 5.20
Xensrver 7.5 unlicensed (free?)
I searched the forum but since no error is produced I'm stuck. Any help would be appreciated.
Just updated to 5.21 (web and server). Still had the same result
Transfer is successful but merge does not start or error out.
I would suggest kicking off a backup job and then start an SSH sesssion where you can monitor the XO logs.
I use this command to monitor the logs --
journalctl -u xo-server -f -n 50
Here is the logs from the last backup I started
Jun 28 12:29:17 vpn xo-server: yarn run v1.7.0
Jun 28 12:29:17 vpn xo-server: $ node bin/xo-server
Jun 28 12:29:22 vpn xo-server: Thu, 28 Jun 2018 16:29:22 GMT app-conf /opt/xen-orchestra/packages/xo-server/config.json
Jun 28 12:29:22 vpn xo-server: Thu, 28 Jun 2018 16:29:22 GMT app-conf /etc/xo-server/config.yaml
Jun 28 12:29:22 vpn xo-server: 2018-06-28T16:29:22.775Z xo:main Configuration loaded.
Jun 28 12:29:22 vpn xo-server: 2018-06-28T16:29:22.780Z xo:main Web server listening on http://[::]:80
Jun 28 12:29:22 vpn xo-server: 2018-06-28T16:29:22.781Z xo:main Group changed to root
Jun 28 12:29:22 vpn xo-server: 2018-06-28T16:29:22.781Z xo:main User changed to root
Jun 28 12:29:22 vpn xo-server: 2018-06-28T16:29:22.842Z xo:hooks start start…
Jun 28 12:29:22 vpn xo-server: 2018-06-28T16:29:22.964Z xo:hooks start finished
Jun 28 12:29:22 vpn xo-server: 2018-06-28T16:29:22.965Z xo:hooks clean start…
Jun 28 12:29:23 vpn xo-server: 2018-06-28T16:29:23.078Z xo:hooks clean finished
Jun 28 12:29:23 vpn xo-server: 2018-06-28T16:29:23.155Z xo:main Setting up / → ../xo-web/dist/
Jun 28 12:29:55 vpn xo-server: 2018-06-28T16:29:55.811Z xo:main + WebSocket connection (::ffff:10.8.0.6)
Jun 28 12:30:28 vpn xo-server: 2018-06-28T16:30:28.940Z xo:xapi Snapshotting VM PBX as [XO Backup Test] PBX
Jun 28 12:30:31 vpn xo-server: 2018-06-28T16:30:31.844Z xo:xapi exporting VDI PBXData (from base PBXData)
Jun 28 12:30:31 vpn xo-server: 2018-06-28T16:30:31.846Z xo:xapi exporting VDI PBXBoot (from base PBXBoot)
Jun 28 12:35:13 vpn xo-server: 2018-06-28T16:35:13.517Z xo:main - WebSocket connection (::ffff:10.8.0.6)
What kind of remote are you using?
@olivierlambert I have tried both NFS and local disk. Both have the same behavior. Is there a way to increase log level to make the steps more verbose?
Anything is Settings/logs?
There is nothing in Settings / Logs
Any update to this issue? As i am also having the same issue while running the community version of XO. Currently running latest xo-server 5.25.2 and xo-web 5.25.0 in ubuntu 16.04. As @mmartinwechu said, there is nothing in log. also journalctl generated below log:
Aug 28 13:28:16 Orchestra xo-server: 2018-08-28T07:43:16.742Z xo:xapi Snapshotting VM MAIL9 as [XO Backup Delta Test] MAIL9
Aug 28 13:28:21 Orchestra xo-server: 2018-08-28T07:43:21.213Z xo:xapi exporting VDI MAIL9 0 (from base MAIL9 0)
Aug 28 13:29:14 Orchestra xo-server: 2018-08-28T07:44:14.931Z xo:perf blocked for 509ms
Also attaching screenshot of the xo-web interface.
I don't think we have any issue reported from XOA customers
@Danp do you experience problems on your side?
@olivierlambert I have installed the xen-orchestra with Jarli01's install script from github when the xo-server version was 5.23 then updated that to 5.24.2 and currently to 5.25.2 today. Also tried with different Xen server versions and XCP-NG latest as well with fresh orchestra installations. Before that tried with installation from source as well with same results. Delta backup seems to work fine until it hits retention number which I have set to 2 for testing.
@Rachendra can you just do quickly a test with a XOA (register for the free trial, it's free and you'll have all features). If you can reproduce, then we'll be sure about this.
@olivierlambert I will try out the free trial and let you know.
@olivierlambert No, not currently experiencing issues w/ merging on delta backups.
@Rachendra What base OS did you use when you created your XO VM?
@Danp using ubuntu 16.04 as base os for xo vm.
@rachendra Suggest that you switch to a more recent version. I'm using 18.04 and not seeing these issues.
@Danp upgraded ubuntu 16.04 to 18.04 still having same issue. So, building new one from scratch. Can you share with me the nodejs and yarn version you used to compile the XO in 18.04? i am suspecting the problem with nodejs. @olivierlambert used trial XOA. Didn't have the issue. Now trying to build new one with nodejs v6 as used in XOA.
Now you know why we only make pro support for XOA installs: the environment can lead to side effects that are very hard to debug… With XOA, everything is in a known env, which facilitate our short fix time.
@rachendra Technically, I'm running Ubuntu 18.04.1. Here's the version info you asked for
$ node -v v8.11.4 $ yarn -v 1.9.4
Have you tried installing XO with one of the existing install scripts, such as this one?