Error Continous Replications 5.18.0



  • hi!
    First try on Production Pool XS 7.1, for example this.

    [{"message":"VDI_IO_ERROR(Device I/O errors)","stack":"XapiError: VDI_IO_ERROR(Device I/O errors)\n at wrapError (/opt/xen-orchestra/packages/xen-api/src/index.js:111:9)\n at getTaskResult (/opt/xen-orchestra/packages/xen-api/src/index.js:189:22)\n at Xapi._addObject (/opt/xen-orchestra/packages/xen-api/src/index.js:796:8)\n at /opt/xen-orchestra/packages/xen-api/src/index.js:832:13\n at arrayEach (/opt/xen-orchestra/node_modules/lodash/_arrayEach.js:15:9)\n at forEach (/opt/xen-orchestra/node_modules/lodash/forEach.js:38:10)\n at Xapi._processEvents (/opt/xen-orchestra/packages/xen-api/src/index.js:827:12)\n at onSuccess (/opt/xen-orchestra/packages/xen-api/src/index.js:850:11)\n at run (/opt/xen-orchestra/node_modules/core-js/modules/es6.promise.js:66:22)\n at /opt/xen-orchestra/node_modules/core-js/modules/es6.promise.js:79:30\n at flush (/opt/xen-orchestra/node_modules/core-js/modules/_microtask.js:18:9)\n at process._tickCallback (internal/process/next_tick.js:112:11)","code":"VDI_IO_ERROR","params":["Device I/O errors"],"url":"https://192.168.222.202/import_raw_vdi/?format=vhd&vdi=OpaqueRef%3A975806dd-0512-48d3-a40b-ac1e69b25778&session_id=OpaqueRef%3A1703e4db-8679-4ec1-a203-afda1b7128c0&task_id=OpaqueRef%3Ad03563a8-1332-4fcc-b036-95e2d1cf5a85"}]

    First try on develop pool
    the error is "interrupted"

    Now I'm trying Backup-legacy, new a few minutes 🙂



  • Well tried again with backup legacy

    none of them has worked.

    Production Pool. (inside the same SR)
    at first I received a time out error, changed the value to 120 seconds and now the error is " VDI_IO_ERROR(Device I/O errors)"
    as much I retried

    Development pool (inside the same SR)
    I'm receiving a "interrupted" error every time I try.



  • Something that fails almost everytime would have been detected if reproducible 😕

    I suspect a XenServer issue, please check your XS logs.



  • but the backup-legacy also fails with XCP-ng 7.4

    where can I see this logs ?



  • XenServer and XCP-ng are 99.99% similar.

    edit: in /var/log/xensource.log and /var/log/SMlog



  • @olivierlambert said in Error Continous Replications 5.18.0:

    /var/log/xensource.log

    thanks.

    do you want me to copy & paste it here ?



  • I'm pretty busy today, and the log could be really big 😕 Try to pinpoint evens that's time related to errors you saw in XO.



  • yes, you're right.
    well I think I catch something

    this is a develop server while doing a backup-legacy, when the backup stoped at 42% more or less

    Apr  6 15:26:34 XCP1 xapi: [debug|XCP1|6 ha_monitor|HA monitor D:bc6ea1becaa8|xapi_ha] Processing warnings
    Apr  6 15:26:34 XCP1 xapi: [debug|XCP1|6 ha_monitor|HA monitor D:bc6ea1becaa8|xapi_ha] Done with warnings
    Apr  6 15:26:34 XCP1 xapi: [debug|XCP1|6 ha_monitor|HA monitor D:bc6ea1becaa8|xapi_ha] The node we think is the master is still alive and marked as master; this is OK
    Apr  6 15:26:42 XCP1 xapi: [error|XCP1|4262 INET :::80|dispatch:session.login_with_password D:c8f25c93214a|backtrace] session.login_with_password D:1f68466fceda failed with exception Server_error(HOST_IS_SLAVE, [ 192.168.222.230 ])
    Apr  6 15:26:42 XCP1 xapi: [error|XCP1|4262 INET :::80|dispatch:session.login_with_password D:c8f25c93214a|backtrace] Raised Server_error(HOST_IS_SLAVE, [ 192.168.222.230 ])
    Apr  6 15:26:42 XCP1 xapi: [error|XCP1|4262 INET :::80|dispatch:session.login_with_password D:c8f25c93214a|backtrace] 1/8 xapi @ XCP1 Raised at file ocaml/xapi/xapi_session.ml, line 383
    Apr  6 15:26:42 XCP1 xapi: [error|XCP1|4262 INET :::80|dispatch:session.login_with_password D:c8f25c93214a|backtrace] 2/8 xapi @ XCP1 Called from file ocaml/xapi/xapi_session.ml, line 39
    Apr  6 15:26:42 XCP1 xapi: [error|XCP1|4262 INET :::80|dispatch:session.login_with_password D:c8f25c93214a|backtrace] 3/8 xapi @ XCP1 Called from file ocaml/xapi/xapi_session.ml, line 39
    Apr  6 15:26:42 XCP1 xapi: [error|XCP1|4262 INET :::80|dispatch:session.login_with_password D:c8f25c93214a|backtrace] 4/8 xapi @ XCP1 Called from file ocaml/xapi/server_helpers.ml, line 69
    Apr  6 15:26:42 XCP1 xapi: [error|XCP1|4262 INET :::80|dispatch:session.login_with_password D:c8f25c93214a|backtrace] 5/8 xapi @ XCP1 Called from file ocaml/xapi/server_helpers.ml, line 91
    Apr  6 15:26:42 XCP1 xapi: [error|XCP1|4262 INET :::80|dispatch:session.login_with_password D:c8f25c93214a|backtrace] 6/8 xapi @ XCP1 Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 22
    Apr  6 15:26:42 XCP1 xapi: [error|XCP1|4262 INET :::80|dispatch:session.login_with_password D:c8f25c93214a|backtrace] 7/8 xapi @ XCP1 Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 26
    Apr  6 15:26:42 XCP1 xapi: [error|XCP1|4262 INET :::80|dispatch:session.login_with_password D:c8f25c93214a|backtrace] 8/8 xapi @ XCP1 Called from file lib/backtrace.ml, line 177
    Apr  6 15:26:42 XCP1 xapi: [error|XCP1|4262 INET :::80|dispatch:session.login_with_password D:c8f25c93214a|backtrace]
    Apr  6 15:26:47 XCP1 xapi: [ info|XCP1|4263 UNIX /var/lib/xcp/xapi||cli] xe host-ha-xapi-healthcheck username=root password=(omitted)
    Apr  6 15:26:47 XCP1 xapi: [debug|XCP1|4263 UNIX /var/lib/xcp/xapi|session.slave_local_login_with_password D:822808a40866|xapi] Add session to local storage
    Apr  6 15:26:49 XCP1 xapi: [debug|XCP1|4261 ||xmlrpc_client] stunnel pid: 18843 (cached = true) returned stunnel to cache
    Apr  6 15:26:49 XCP1 xapi: [debug|XCP1|4264 ||mscgen] xapi=>xapi [label="event.from"];
    Apr  6 15:26:49 XCP1 xapi: [debug|XCP1|4264 ||xmlrpc_client] stunnel pid: 14677 (cached = true) connected to 192.168.222.230:443
    Apr  6 15:26:49 XCP1 xapi: [debug|XCP1|4264 ||xmlrpc_client] with_recorded_stunnelpid task_opt=None s_pid=14677
    Apr  6 15:26:54 XCP1 xapi: [debug|XCP1|6 ha_monitor|HA monitor D:bc6ea1becaa8|xapi_ha] Liveset: online 11d55d91-2e0d-4008-803d-42db55ca4cf7 [ L  A ]; 55185571-6dfa-429a-9415-2dacb9ff1f3a [*L  A ]; a16383be-39c6-4bba-8709-75653eacd759 [ LM A ];
    Apr  6 15:26:54 XCP1 xapi: [debug|XCP1|6 ha_monitor|HA monitor D:bc6ea1becaa8|xapi_ha] Processing warnings
    Apr  6 15:26:54 XCP1 xapi: [debug|XCP1|6 ha_monitor|HA monitor D:bc6ea1becaa8|xapi_ha] Done with warnings
    Apr  6 15:26:54 XCP1 xapi: [debug|XCP1|6 ha_monitor|HA monitor D:bc6ea1becaa8|xapi_ha] The node we think is the master is still alive and marked as master; this is OK
    

    copy and paste on another editor, big screen is more legible.

    and now I can see this on tasks
    0_1523017912310_Selecció_054.png



  • It seems you have issue with HA, can you disable it?



  • yes, of course
    on develop pool
    xe pool-ha-disable

    tried again the backup Continuous replication backup-legacy.

    received the same error "interrupted"

    the log:

    Apr  6 15:56:17 XCP1 xapi: [debug|XCP1|4367 INET :::80|handler:http/rrd_updates D:0b2544c2a79f|xmlrpc_client] stunnel pid: 16092 (cached = true) connected to 192.168.222.230:443
    Apr  6 15:56:17 XCP1 xapi: [debug|XCP1|4367 INET :::80|handler:http/rrd_updates D:0b2544c2a79f|xmlrpc_client] with_recorded_stunnelpid task_opt=None s_pid=16092
    Apr  6 15:56:17 XCP1 xapi: [debug|XCP1|4367 INET :::80|handler:http/rrd_updates D:0b2544c2a79f|xmlrpc_client] stunnel pid: 16092 (cached = true) returned stunnel to cache
    Apr  6 15:56:17 XCP1 xapi: [debug|XCP1|4367 INET :::80|Get RRD updates. D:df56aef76915|xapi] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
    Apr  6 15:56:32 XCP1 xapi: [error|XCP1|4368 INET :::80|dispatch:session.login_with_password D:4f8aeaec976e|backtrace] session.login_with_password D:bd4561f87253 failed with exception Server_error(HOST_IS_SLAVE, [ 192.168.222.230 ])
    Apr  6 15:56:32 XCP1 xapi: [error|XCP1|4368 INET :::80|dispatch:session.login_with_password D:4f8aeaec976e|backtrace] Raised Server_error(HOST_IS_SLAVE, [ 192.168.222.230 ])
    Apr  6 15:56:32 XCP1 xapi: [error|XCP1|4368 INET :::80|dispatch:session.login_with_password D:4f8aeaec976e|backtrace] 1/8 xapi @ XCP1 Raised at file ocaml/xapi/xapi_session.ml, line 383
    Apr  6 15:56:32 XCP1 xapi: [error|XCP1|4368 INET :::80|dispatch:session.login_with_password D:4f8aeaec976e|backtrace] 2/8 xapi @ XCP1 Called from file ocaml/xapi/xapi_session.ml, line 39
    Apr  6 15:56:32 XCP1 xapi: [error|XCP1|4368 INET :::80|dispatch:session.login_with_password D:4f8aeaec976e|backtrace] 3/8 xapi @ XCP1 Called from file ocaml/xapi/xapi_session.ml, line 39
    Apr  6 15:56:32 XCP1 xapi: [error|XCP1|4368 INET :::80|dispatch:session.login_with_password D:4f8aeaec976e|backtrace] 4/8 xapi @ XCP1 Called from file ocaml/xapi/server_helpers.ml, line 69
    Apr  6 15:56:32 XCP1 xapi: [error|XCP1|4368 INET :::80|dispatch:session.login_with_password D:4f8aeaec976e|backtrace] 5/8 xapi @ XCP1 Called from file ocaml/xapi/server_helpers.ml, line 91
    Apr  6 15:56:32 XCP1 xapi: [error|XCP1|4368 INET :::80|dispatch:session.login_with_password D:4f8aeaec976e|backtrace] 6/8 xapi @ XCP1 Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 22
    Apr  6 15:56:32 XCP1 xapi: [error|XCP1|4368 INET :::80|dispatch:session.login_with_password D:4f8aeaec976e|backtrace] 7/8 xapi @ XCP1 Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 26
    Apr  6 15:56:32 XCP1 xapi: [error|XCP1|4368 INET :::80|dispatch:session.login_with_password D:4f8aeaec976e|backtrace] 8/8 xapi @ XCP1 Called from file lib/backtrace.ml, line 177
    Apr  6 15:56:32 XCP1 xapi: [error|XCP1|4368 INET :::80|dispatch:session.login_with_password D:4f8aeaec976e|backtrace]
    Apr  6 15:56:35 XCP1 xapi: [debug|XCP1|63 heartbeat|Heartbeat D:b70ceab1b744|mscgen] xapi=>xapi [label="host.tickle_heartbeat"];
    Apr  6 15:56:35 XCP1 xapi: [debug|XCP1|63 heartbeat|Heartbeat D:b70ceab1b744|stunnel] stunnel start
    Apr  6 15:56:35 XCP1 xapi: [debug|XCP1|63 heartbeat|Heartbeat D:b70ceab1b744|xmlrpc_client] stunnel pid: 29155 (cached = false) connected to 192.168.222.230:443
    Apr  6 15:56:35 XCP1 xapi: [debug|XCP1|63 heartbeat|Heartbeat D:b70ceab1b744|xmlrpc_client] with_recorded_stunnelpid task_opt=None s_pid=29155
    


  • Please use three backtick around your text, otherwise it's hard to read it



  • Sorry. Didn't know that. Corrected



  • Can you double check that you connect to this pool with only one server in XOA?



  • @olivierlambert hi. What do you mean? Only 1 XO? At this moment I have 2 XO (one created today i began with problems 3 days ago when first installed) in each pool. Also I connect with XenCenter.

    Or do you mean in Settgins->servers ?
    in this option, I have 2 servers... and I have 3 in the pool, gonna add this one and check.
    or should only be 1 server connected ?



  • No I mean, in the "Settings/Server" view. You should have only 1 server added for one pool.



  • @olivierlambert thanks for your patience, this is what I have now there , the three servers of the pool. its only needed one here ? which one ? 0_1523031460864_Selección_012.jpg



  • You need to ONLY have the pool master. Remove the others and restart xo-server.



  • ok I removed all of them, only master remained. restarted EVERYTHING.
    tested again, and de continuous replication fails.
    in one specific vm, i can see that exporting is at 100% and importing stops at 42%, then it stops wit error. this time is VDI_IO_ERROR(Device I/O errors)



  • well, may be its a problem on my side, I will keep trying.
    If I find it I will tell you.
    Thanks for your time !



  • @bsastre Have you tried this where both hosts are running the same version of XS?


Log in to reply