Error Continous Replications 5.18.0



  • Something that fails almost everytime would have been detected if reproducible 😕

    I suspect a XenServer issue, please check your XS logs.



  • but the backup-legacy also fails with XCP-ng 7.4

    where can I see this logs ?



  • XenServer and XCP-ng are 99.99% similar.

    edit: in /var/log/xensource.log and /var/log/SMlog



  • @olivierlambert said in Error Continous Replications 5.18.0:

    /var/log/xensource.log

    thanks.

    do you want me to copy & paste it here ?



  • I'm pretty busy today, and the log could be really big 😕 Try to pinpoint evens that's time related to errors you saw in XO.



  • yes, you're right.
    well I think I catch something

    this is a develop server while doing a backup-legacy, when the backup stoped at 42% more or less

    Apr  6 15:26:34 XCP1 xapi: [debug|XCP1|6 ha_monitor|HA monitor D:bc6ea1becaa8|xapi_ha] Processing warnings
    Apr  6 15:26:34 XCP1 xapi: [debug|XCP1|6 ha_monitor|HA monitor D:bc6ea1becaa8|xapi_ha] Done with warnings
    Apr  6 15:26:34 XCP1 xapi: [debug|XCP1|6 ha_monitor|HA monitor D:bc6ea1becaa8|xapi_ha] The node we think is the master is still alive and marked as master; this is OK
    Apr  6 15:26:42 XCP1 xapi: [error|XCP1|4262 INET :::80|dispatch:session.login_with_password D:c8f25c93214a|backtrace] session.login_with_password D:1f68466fceda failed with exception Server_error(HOST_IS_SLAVE, [ 192.168.222.230 ])
    Apr  6 15:26:42 XCP1 xapi: [error|XCP1|4262 INET :::80|dispatch:session.login_with_password D:c8f25c93214a|backtrace] Raised Server_error(HOST_IS_SLAVE, [ 192.168.222.230 ])
    Apr  6 15:26:42 XCP1 xapi: [error|XCP1|4262 INET :::80|dispatch:session.login_with_password D:c8f25c93214a|backtrace] 1/8 xapi @ XCP1 Raised at file ocaml/xapi/xapi_session.ml, line 383
    Apr  6 15:26:42 XCP1 xapi: [error|XCP1|4262 INET :::80|dispatch:session.login_with_password D:c8f25c93214a|backtrace] 2/8 xapi @ XCP1 Called from file ocaml/xapi/xapi_session.ml, line 39
    Apr  6 15:26:42 XCP1 xapi: [error|XCP1|4262 INET :::80|dispatch:session.login_with_password D:c8f25c93214a|backtrace] 3/8 xapi @ XCP1 Called from file ocaml/xapi/xapi_session.ml, line 39
    Apr  6 15:26:42 XCP1 xapi: [error|XCP1|4262 INET :::80|dispatch:session.login_with_password D:c8f25c93214a|backtrace] 4/8 xapi @ XCP1 Called from file ocaml/xapi/server_helpers.ml, line 69
    Apr  6 15:26:42 XCP1 xapi: [error|XCP1|4262 INET :::80|dispatch:session.login_with_password D:c8f25c93214a|backtrace] 5/8 xapi @ XCP1 Called from file ocaml/xapi/server_helpers.ml, line 91
    Apr  6 15:26:42 XCP1 xapi: [error|XCP1|4262 INET :::80|dispatch:session.login_with_password D:c8f25c93214a|backtrace] 6/8 xapi @ XCP1 Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 22
    Apr  6 15:26:42 XCP1 xapi: [error|XCP1|4262 INET :::80|dispatch:session.login_with_password D:c8f25c93214a|backtrace] 7/8 xapi @ XCP1 Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 26
    Apr  6 15:26:42 XCP1 xapi: [error|XCP1|4262 INET :::80|dispatch:session.login_with_password D:c8f25c93214a|backtrace] 8/8 xapi @ XCP1 Called from file lib/backtrace.ml, line 177
    Apr  6 15:26:42 XCP1 xapi: [error|XCP1|4262 INET :::80|dispatch:session.login_with_password D:c8f25c93214a|backtrace]
    Apr  6 15:26:47 XCP1 xapi: [ info|XCP1|4263 UNIX /var/lib/xcp/xapi||cli] xe host-ha-xapi-healthcheck username=root password=(omitted)
    Apr  6 15:26:47 XCP1 xapi: [debug|XCP1|4263 UNIX /var/lib/xcp/xapi|session.slave_local_login_with_password D:822808a40866|xapi] Add session to local storage
    Apr  6 15:26:49 XCP1 xapi: [debug|XCP1|4261 ||xmlrpc_client] stunnel pid: 18843 (cached = true) returned stunnel to cache
    Apr  6 15:26:49 XCP1 xapi: [debug|XCP1|4264 ||mscgen] xapi=>xapi [label="event.from"];
    Apr  6 15:26:49 XCP1 xapi: [debug|XCP1|4264 ||xmlrpc_client] stunnel pid: 14677 (cached = true) connected to 192.168.222.230:443
    Apr  6 15:26:49 XCP1 xapi: [debug|XCP1|4264 ||xmlrpc_client] with_recorded_stunnelpid task_opt=None s_pid=14677
    Apr  6 15:26:54 XCP1 xapi: [debug|XCP1|6 ha_monitor|HA monitor D:bc6ea1becaa8|xapi_ha] Liveset: online 11d55d91-2e0d-4008-803d-42db55ca4cf7 [ L  A ]; 55185571-6dfa-429a-9415-2dacb9ff1f3a [*L  A ]; a16383be-39c6-4bba-8709-75653eacd759 [ LM A ];
    Apr  6 15:26:54 XCP1 xapi: [debug|XCP1|6 ha_monitor|HA monitor D:bc6ea1becaa8|xapi_ha] Processing warnings
    Apr  6 15:26:54 XCP1 xapi: [debug|XCP1|6 ha_monitor|HA monitor D:bc6ea1becaa8|xapi_ha] Done with warnings
    Apr  6 15:26:54 XCP1 xapi: [debug|XCP1|6 ha_monitor|HA monitor D:bc6ea1becaa8|xapi_ha] The node we think is the master is still alive and marked as master; this is OK
    

    copy and paste on another editor, big screen is more legible.

    and now I can see this on tasks
    0_1523017912310_Selecció_054.png



  • It seems you have issue with HA, can you disable it?



  • yes, of course
    on develop pool
    xe pool-ha-disable

    tried again the backup Continuous replication backup-legacy.

    received the same error "interrupted"

    the log:

    Apr  6 15:56:17 XCP1 xapi: [debug|XCP1|4367 INET :::80|handler:http/rrd_updates D:0b2544c2a79f|xmlrpc_client] stunnel pid: 16092 (cached = true) connected to 192.168.222.230:443
    Apr  6 15:56:17 XCP1 xapi: [debug|XCP1|4367 INET :::80|handler:http/rrd_updates D:0b2544c2a79f|xmlrpc_client] with_recorded_stunnelpid task_opt=None s_pid=16092
    Apr  6 15:56:17 XCP1 xapi: [debug|XCP1|4367 INET :::80|handler:http/rrd_updates D:0b2544c2a79f|xmlrpc_client] stunnel pid: 16092 (cached = true) returned stunnel to cache
    Apr  6 15:56:17 XCP1 xapi: [debug|XCP1|4367 INET :::80|Get RRD updates. D:df56aef76915|xapi] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
    Apr  6 15:56:32 XCP1 xapi: [error|XCP1|4368 INET :::80|dispatch:session.login_with_password D:4f8aeaec976e|backtrace] session.login_with_password D:bd4561f87253 failed with exception Server_error(HOST_IS_SLAVE, [ 192.168.222.230 ])
    Apr  6 15:56:32 XCP1 xapi: [error|XCP1|4368 INET :::80|dispatch:session.login_with_password D:4f8aeaec976e|backtrace] Raised Server_error(HOST_IS_SLAVE, [ 192.168.222.230 ])
    Apr  6 15:56:32 XCP1 xapi: [error|XCP1|4368 INET :::80|dispatch:session.login_with_password D:4f8aeaec976e|backtrace] 1/8 xapi @ XCP1 Raised at file ocaml/xapi/xapi_session.ml, line 383
    Apr  6 15:56:32 XCP1 xapi: [error|XCP1|4368 INET :::80|dispatch:session.login_with_password D:4f8aeaec976e|backtrace] 2/8 xapi @ XCP1 Called from file ocaml/xapi/xapi_session.ml, line 39
    Apr  6 15:56:32 XCP1 xapi: [error|XCP1|4368 INET :::80|dispatch:session.login_with_password D:4f8aeaec976e|backtrace] 3/8 xapi @ XCP1 Called from file ocaml/xapi/xapi_session.ml, line 39
    Apr  6 15:56:32 XCP1 xapi: [error|XCP1|4368 INET :::80|dispatch:session.login_with_password D:4f8aeaec976e|backtrace] 4/8 xapi @ XCP1 Called from file ocaml/xapi/server_helpers.ml, line 69
    Apr  6 15:56:32 XCP1 xapi: [error|XCP1|4368 INET :::80|dispatch:session.login_with_password D:4f8aeaec976e|backtrace] 5/8 xapi @ XCP1 Called from file ocaml/xapi/server_helpers.ml, line 91
    Apr  6 15:56:32 XCP1 xapi: [error|XCP1|4368 INET :::80|dispatch:session.login_with_password D:4f8aeaec976e|backtrace] 6/8 xapi @ XCP1 Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 22
    Apr  6 15:56:32 XCP1 xapi: [error|XCP1|4368 INET :::80|dispatch:session.login_with_password D:4f8aeaec976e|backtrace] 7/8 xapi @ XCP1 Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 26
    Apr  6 15:56:32 XCP1 xapi: [error|XCP1|4368 INET :::80|dispatch:session.login_with_password D:4f8aeaec976e|backtrace] 8/8 xapi @ XCP1 Called from file lib/backtrace.ml, line 177
    Apr  6 15:56:32 XCP1 xapi: [error|XCP1|4368 INET :::80|dispatch:session.login_with_password D:4f8aeaec976e|backtrace]
    Apr  6 15:56:35 XCP1 xapi: [debug|XCP1|63 heartbeat|Heartbeat D:b70ceab1b744|mscgen] xapi=>xapi [label="host.tickle_heartbeat"];
    Apr  6 15:56:35 XCP1 xapi: [debug|XCP1|63 heartbeat|Heartbeat D:b70ceab1b744|stunnel] stunnel start
    Apr  6 15:56:35 XCP1 xapi: [debug|XCP1|63 heartbeat|Heartbeat D:b70ceab1b744|xmlrpc_client] stunnel pid: 29155 (cached = false) connected to 192.168.222.230:443
    Apr  6 15:56:35 XCP1 xapi: [debug|XCP1|63 heartbeat|Heartbeat D:b70ceab1b744|xmlrpc_client] with_recorded_stunnelpid task_opt=None s_pid=29155
    


  • Please use three backtick around your text, otherwise it's hard to read it



  • Sorry. Didn't know that. Corrected



  • Can you double check that you connect to this pool with only one server in XOA?



  • @olivierlambert hi. What do you mean? Only 1 XO? At this moment I have 2 XO (one created today i began with problems 3 days ago when first installed) in each pool. Also I connect with XenCenter.

    Or do you mean in Settgins->servers ?
    in this option, I have 2 servers... and I have 3 in the pool, gonna add this one and check.
    or should only be 1 server connected ?



  • No I mean, in the "Settings/Server" view. You should have only 1 server added for one pool.



  • @olivierlambert thanks for your patience, this is what I have now there , the three servers of the pool. its only needed one here ? which one ? 0_1523031460864_Selección_012.jpg



  • You need to ONLY have the pool master. Remove the others and restart xo-server.



  • ok I removed all of them, only master remained. restarted EVERYTHING.
    tested again, and de continuous replication fails.
    in one specific vm, i can see that exporting is at 100% and importing stops at 42%, then it stops wit error. this time is VDI_IO_ERROR(Device I/O errors)



  • well, may be its a problem on my side, I will keep trying.
    If I find it I will tell you.
    Thanks for your time !



  • @bsastre Have you tried this where both hosts are running the same version of XS?



  • hi @Danp yes, I've done a lot of combinations.

    nevermind, maybe in summer I will try to reinstall everything from scratch and try again.

    thanks again for your time and effort.



  • well tried again.

    but this time with a XS 7.1 instead XCP 7.4

    there are 3 servers
    192.168.222.210
    192.168.222.220
    192.168.222.230

    192.168.222.210 is the master
    192.168.222.220 is the host that holds de VM being CR.

    in XO there is only one server connected at the settings 192.168.222.210 (master)

     [{"message":"VDI_IO_ERROR(Device I/O errors)","stack":"XapiError: VDI_IO_ERROR(Device I/O errors)\n at wrapError (/opt/xen-orchestra/packages/xen-api/src/index.js:111:9)\n at getTaskResult (/opt/xen-orchestra/packages/xen-api/src/index.js:191:26)\n at Xapi._addObject (/opt/xen-orchestra/packages/xen-api/src/index.js:797:23)\n at /opt/xen-orchestra/packages/xen-api/src/index.js:835:13\n at arrayEach (/opt/xen-orchestra/node_modules/lodash/_arrayEach.js:15:9)\n at forEach (/opt/xen-orchestra/node_modules/lodash/forEach.js:38:10)\n at Xapi._processEvents (/opt/xen-orchestra/packages/xen-api/src/index.js:830:12)\n at onSuccess (/opt/xen-orchestra/packages/xen-api/src/index.js:853:11)\n at run (/opt/xen-orchestra/node_modules/core-js/modules/es6.promise.js:66:22)\n at /opt/xen-orchestra/node_modules/core-js/modules/es6.promise.js:83:30\n at flush (/opt/xen-orchestra/node_modules/core-js/modules/_microtask.js:18:9)\n at process._tickCallback (internal/process/next_tick.js:112:11)","code":"VDI_IO_ERROR","params":["Device I/O errors"],"url":"https://192.168.222.230/import_raw_vdi/?format=vhd&vdi=OpaqueRef%3Abedee949-add4-2d80-57b3-b525b17ae751&session_id=OpaqueRef%3Aa855be20-04e6-8b24-04d9-8133b0ec4bde&task_id=OpaqueRef%3A7c1febb8-07f8-e9e6-6b1e-56bdd174a324"}]
    

Log in to reply