Attachment 924534 Details for Bug 1122568 – Annotated sanlock log on a host that became non operational

Annotated sanlock log on a host that became non operational

annotated-sanlock.log (text/x-log), 3.82 KB, created by Nir Soffer on 2014-08-06 16:58:32 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Nir Soffer

Created: 2014-08-06 16:58:32 UTC

Size: 3.82 KB

patch

obsolete

>Last succesfull sanlock opoeration with storage:
>
>2014-08-06 13:56:46+0000 1490039 [2171]: s117:r34 resource 68957f61-33a4-47ea-9b7d-4e0a84639841:SDM:/rhev/data-center/mnt/cmplnt-nas01.scale.openstack.engineering.redhat.com:_rhev__real_data_storage__mount/68957f61-33a4-47ea-9b7d-4e0a84639841/dom_md/leases:1048576 for 21,31,24679
>
>sanlock cannot access storage:
>
>2014-08-06 14:19:24+0000 1491397 [29509]: 68957f61 aio timeout 0 0x7f91d80008c0:0x7f91d80008d0:0x7f91e4e23000 ioto 10 to_count 1
>2014-08-06 14:19:24+0000 1491397 [29509]: s117 delta_renew read rv -202 offset 0 /rhev/data-center/mnt/cmplnt-nas01.scale.openstack.engineering.redhat.com:_rhev__real_data_storage__mount/68957f61-33a4-47ea-9b7d-4e0a84639841/dom_md/ids
>2014-08-06 14:19:24+0000 1491397 [29509]: s117 renewal error -202 delta_length 10 last_success 1491367
>2014-08-06 14:19:35+0000 1491408 [29509]: 68957f61 aio timeout 0 0x7f91d8000910:0x7f91d8000920:0x7f91e0e83000 ioto 10 to_count 2
>2014-08-06 14:19:35+0000 1491408 [29509]: s117 delta_renew read rv -202 offset 0 /rhev/data-center/mnt/cmplnt-nas01.scale.openstack.engineering.redhat.com:_rhev__real_data_storage__mount/68957f61-33a4-47ea-9b7d-4e0a84639841/dom_md/ids
>2014-08-06 14:19:35+0000 1491408 [29509]: s117 renewal error -202 delta_length 10 last_success 1491367
>2014-08-06 14:19:46+0000 1491419 [29509]: 68957f61 aio timeout 0 0x7f91d8000960:0x7f91d8000970:0x7f91e0c80000 ioto 10 to_count 3
>2014-08-06 14:19:46+0000 1491419 [29509]: s117 delta_renew read rv -202 offset 0 /rhev/data-center/mnt/cmplnt-nas01.scale.openstack.engineering.redhat.com:_rhev__real_data_storage__mount/68957f61-33a4-47ea-9b7d-4e0a84639841/dom_md/ids
>2014-08-06 14:19:46+0000 1491419 [29509]: s117 renewal error -202 delta_length 10 last_success 1491367
>2014-08-06 14:19:54+0000 1491427 [2165]: s117 check_our_lease warning 60 last_success 1491367
>...
>2014-08-06 14:20:13+0000 1491446 [2165]: s117 check_our_lease warning 79 last_success 1491367
>
>Sanlock failed ot access storage for 80 seconds:
>
>2014-08-06 14:20:14+0000 1491447 [2165]: s117 check_our_lease failed 80
>
>Sanlock trying to terminate vdsm:
>
>2014-08-06 14:20:14+0000 1491447 [2165]: s117 kill 24679 sig 15 count 1
>...
>2014-08-06 14:20:53+0000 1491486 [2165]: s117 kill 24679 sig 15 count 40
>
>vdsm did not exit after 40 retries (need to check way).
>
>Sanlock kills vdsm:
>
>2014-08-06 14:20:54+0000 1491487 [2165]: s117 kill 24679 sig 9 count 41
>
>vdsm was killed after 41 seconds:
>
>2014-08-06 14:20:54+0000 1491487 [2165]: dead 24679 ci 21 count 41
>
>Sanlock still cannot access storage:
>
>2014-08-06 14:21:01+0000 1491494 [29509]: s117 delta_renew read rv -2 offset 0 /rhev/data-center/mnt/cmplnt-nas01.scale.openstack.engineering.redhat.com:_rhev__real_data_storage__mount/68957f61-33a4-47ea-9b7d-4e0a84639841/dom_md/ids
>2014-08-06 14:21:01+0000 1491494 [29509]: s117 renewal error -2 delta_length 10 last_success 1491367
>2014-08-06 14:21:11+0000 1491504 [29509]: 68957f61 close_task_aio 0 0x7f91d80008c0 busy
>...
>2014-08-06 14:24:41+0000 1491714 [29509]: 68957f61 close_task_aio 3 0x7f91d80009b0 busy
>2014-08-06 14:24:43+0000 1491716 [29509]: 68957f61 aio collect 0x7f91d80008c0:0x7f91d80008d0:0x7f91e4e23000 result 1048576:0 close free
>...
>2014-08-06 14:24:43+0000 1491716 [29509]: 68957f61 aio collect 0x7f91d80009b0:0x7f91d80009c0:0x7f91e0b7e000 result 1048576:0 close free
>
>Sanlock can access storage now:
>
>2014-08-06 14:25:42+0000 1491775 [2171]: s118 lockspace 68957f61-33a4-47ea-9b7d-4e0a84639841:22:/rhev/data-center/mnt/cmplnt-nas01.scale.openstack.engineering.redhat.com:_rhev__real_data_storage__mount/68957f61-33a4-47ea-9b7d-4e0a84639841/dom_md/ids:0
>2014-08-06 14:26:03+0000 1491796 [2170]: s118:r35 resource 68957f61-33a4-47ea-9b7d-4e0a84639841:SDM:/rhev/data-center/mnt/cmplnt-nas01.scale.openstack.engineering.redhat.com:_rhev__real_data_storage__mount/68957f61-33a4-47ea-9b7d-4e0a84639841/dom_md/leases:1048576 for 21,31,29777
>

Actions: View

Attachments on bug 1122568: 920231 | 920233 | 920234 | 921458 | 921487 | 921720 | 922176 | 924524 | 924526 | 924529 | 924534 | 928373