Bug 1627613

Summary: After esx hosts rebooted , all path status is unused. the rebooted hosts can't use the iscsi storage.
Product: [Red Hat Storage] Red Hat Ceph Storage Reporter: liuwei <wliu>
Component: iSCSIAssignee: Mike Christie <mchristi>
Status: CLOSED DUPLICATE QA Contact: Madhavi Kasturi <mkasturi>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 3.0CC: assingh, ceph-eng-bugs, ceph-qe-bugs, mchristi
Target Milestone: rc   
Target Release: 3.*   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-09-12 15:29:09 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description liuwei 2018-09-11 03:48:15 UTC 
Description of problem:

After esx hosts rebooted ,  all path status is unused.    the  hosts can't use the iscsi storage.

From the tcmu-runner log checking, the error is below:

For iscigw node osd16.

2018-09-05 12:59:31.656 1702 [ERROR] tcmu_notify_lock_lost:198 rbd/rbd.ceph-iscsi-gw-lun2100: Async lock drop. Old state 1
2018-09-05 12:59:32.554 1702 [WARN] tcmu_rbd_lock:446 rbd/rbd.ceph-iscsi-gw-lun2100: Acquired exclusive lock.
2018-09-05 12:59:35.766 1702 [ERROR] tcmu_notify_lock_lost:198 rbd/rbd.ceph-iscsi-gw-lun2106: Async lock drop. Old state 1
2018-09-05 12:59:36.558 1702 [WARN] tcmu_rbd_lock:446 rbd/rbd.ceph-iscsi-gw-lun2106: Acquired exclusive lock.
2018-09-05 12:59:36.881 1702 [ERROR] tcmu_notify_lock_lost:198 rbd/rbd.ceph-iscsi-gw-lun2107: Async lock drop. Old state 1
2018-09-05 12:59:37.645 1702 [WARN] tcmu_rbd_lock:446 rbd/rbd.ceph-iscsi-gw-lun2107: Acquired exclusive lock.
2018-09-06 19:20:38.979 1702 [ERROR] tcmu_notify_lock_lost:198 rbd/rbd.ceph-iscsi-gw-lun2200: Async lock drop. Old state 1
2018-09-06 19:20:40.119 1702 [WARN] tcmu_rbd_lock:446 rbd/rbd.ceph-iscsi-gw-lun2200: Acquired exclusive lock.
2018-09-06 19:35:42.209 1702 [ERROR] tcmu_notify_lock_lost:198 rbd/rbd.ceph-iscsi-gw-lun2200: Async lock drop. Old state 1
2018-09-06 19:35:44.325 1702 [WARN] tcmu_rbd_lock:446 rbd/rbd.ceph-iscsi-gw-lun2200: Acquired exclusive lock.
2018-09-06 19:56:58.534 1702 [ERROR] tcmu_notify_lock_lost:198 rbd/rbd.ceph-iscsi-gw-lun2200: Async lock drop. Old state 1
2018-09-06 19:57:00.441 1702 [WARN] tcmu_rbd_lock:446 rbd/rbd.ceph-iscsi-gw-lun2200: Acquired exclusive lock.


For osd24

2018-09-06 19:20:33.710 1687 [WARN] tcmu_rbd_lock:446 rbd/rbd.ceph-iscsi-gw-lun2200: Acquired exclusive lock.
2018-09-06 19:35:35.297 1687 [ERROR] tcmu_notify_lock_lost:198 rbd/rbd.ceph-iscsi-gw-lun2200: Async lock drop. Old state 1
2018-09-06 19:35:36.245 1687 [WARN] tcmu_rbd_lock:446 rbd/rbd.ceph-iscsi-gw-lun2200: Acquired exclusive lock.
2018-09-06 19:56:50.649 1687 [ERROR] tcmu_notify_lock_lost:198 rbd/rbd.ceph-iscsi-gw-lun2200: Async lock drop. Old state 1
2018-09-06 19:56:52.485 1687 [WARN] tcmu_rbd_lock:446 rbd/rbd.ceph-iscsi-gw-lun2200: Acquired exclusive lock.
2018-09-07 09:40:46.836 1687 [ERROR] tcmu_rbd_lock_break:397 rbd/rbd.ceph-iscsi-gw-lun3505: Could not break lock from 10.3.130.231:0/2540389405. (Err -16)
2018-09-07 09:40:47.837 1687 [ERROR] tcmu_rbd_lock_break:397 rbd/rbd.ceph-iscsi-gw-lun3505: Could not break lock from 10.3.130.231:0/2540389405. (Err -16)
2018-09-07 09:40:48.838 1687 [ERROR] tcmu_rbd_lock_break:397 rbd/rbd.ceph-iscsi-gw-lun3505: Could not break lock from 10.3.130.231:0/2540389405. (Err -16)
2018-09-07 09:40:49.840 1687 [ERROR] tcmu_rbd_lock_break:397 rbd/rbd.ceph-iscsi-gw-lun3505: Could not break lock from 10.3.130.231:0/2540389405. (Err -16)
2018-09-07 09:40:50.841 1687 [ERROR] tcmu_rbd_lock_break:397 rbd/rbd.ceph-iscsi-gw-lun3505: Could not break lock from 10.3.130.231:0/2540389405. (Err -16)
2018-09-07 09:40:52.368 1687 [ERROR] tcmu_rbd_lock_break:397 rbd/rbd.ceph-iscsi-gw-lun3505: Could not break lock from 10.3.130.231:0/2540389405. (Err -16)
2018-09-07 09:40:53.369 1687 [ERROR] tcmu_rbd_lock_break:397 rbd/rbd.ceph-iscsi-gw-lun3505: Could not break lock from 10.3.130.231:0/2540389405. (Err -16)
2018-09-07 09:40:54.370 1687 [ERROR] tcmu_rbd_lock_break:397 rbd/rbd.ceph-iscsi-gw-lun3505: Could not break lock from 10.3.130.231:0/2540389405. (Err -16)
2018-09-07 09:40:55.371 1687 [ERROR] tcmu_rbd_lock_break:397 rbd/rbd.ceph-iscsi-gw-lun3505: Could not break lock from 10.3.130.231:0/2540389405. (Err -16)
2018-09-07 09:40:56.372 1687 [ERROR] tcmu_rbd_lock_break:397 rbd/rbd.ceph-iscsi-gw-lun3505: Could not break lock from 10.3.130.231:0/2540389405. (Err -16)
2018-09-07 09:40:58.368 1687 [ERROR] tcmu_rbd_lock_break:397 rbd/rbd.ceph-iscsi-gw-lun3505: Could not break lock from 10.3.130.231:0/2540389405. (Err -16)
2018-09-07 09:40:59.369 1687 [ERROR] tcmu_rbd_lock_break:397 rbd/rbd.ceph-iscsi-gw-lun3505: Could not break lock from 10.3.130.231:0/2540389405. (Err -16)
2018-09-07 09:41:00.370 1687 [ERROR] tcmu_rbd_lock_break:397 rbd/rbd.ceph-iscsi-gw-lun3505: Could not break lock from 10.3.130.231:0/2540389405. (Err -16)
2018-09-07 09:41:01.371 1687 [ERROR] tcmu_rbd_lock_break:397 rbd/rbd.ceph-iscsi-gw-lun3505: Could not break lock from 10.3.130.231:0/2540389405. (Err -16)
2018-09-07 09:41:02.372 1687 [ERROR] tcmu_rbd_lock_break:397 rbd/rbd.ceph-iscsi-gw-lun3505: Could not break lock from 10.3.130.231:0/2540389405. (Err -16)
2018-09-07 09:41:04.368 1687 [ERROR] tcmu_rbd_lock_break:397 rbd/rbd.ceph-iscsi-gw-lun3505: Could not break lock from 10.3.130.231:0/2540389405. (Err -16)
2018-09-07 09:41:05.369 1687 [ERROR] tcmu_rbd_lock_break:397 rbd/rbd.ceph-iscsi-gw-lun3505: Could not break lock from 10.3.130.231:0/2540389405. (Err -16)
2018-09-07 09:41:06.370 1687 [ERROR] tcmu_rbd_lock_break:397 rbd/rbd.ceph-iscsi-gw-lun3505: Could not break lock from 10.3.130.231:0/2540389405. (Err -16)
2018-09-07 09:41:07.371 1687 [ERROR] tcmu_rbd_lock_break:397 rbd/rbd.ceph-iscsi-gw-lun3505: Could not break lock from 10.3.130.231:0/2540389405. (Err -16)
2018-09-07 09:41:08.372 1687 [ERROR] tcmu_rbd_lock_break:397 rbd/rbd.ceph-iscsi-gw-lun3505: Could not break lock from 10.3.130.231:0/2540389405. (Err -16)
2018-09-07 09:41:10.368 1687 [ERROR] tcmu_rbd_lock_break:397 rbd/rbd.ceph-iscsi-gw-lun3505: Could not break lock from 10.3.130.231:0/2540389405. (Err -16)
2018-09-07 09:41:11.369 1687 [ERROR] tcmu_rbd_lock_break:397 rbd/rbd.ceph-iscsi-gw-lun3505: Could not break lock from 10.3.130.231:0/2540389405. (Err -16)
2018-09-07 09:41:12.370 1687 [ERROR] tcmu_rbd_lock_break:397 rbd/rbd.ceph-iscsi-gw-lun3505: Could not break lock from 10.3.130.231:0/2540389405. (Err -16)
2018-09-07 09:41:13.371 1687 [ERROR] tcmu_rbd_lock_break:397 rbd/rbd.ceph-iscsi-gw-lun3505: Could not break lock from 10.3.130.231:0/2540389405. (Err -16)
2018-09-07 09:41:14.372 1687 [ERROR] tcmu_rbd_lock_break:397 rbd/rbd.ceph-iscsi-gw-lun3505: Could not break lock from 10.3.130.231:0/2540389405. (Err -16)




Version-Release number of selected component (if applicable):

 
ceph-common-12.2.4-10.el7cp.x86_64                          Sat Jun 16 02:08:00 2018
ceph-iscsi-cli-2.5-10.el7cp.noarch                          Sat Jun 16 08:38:12 2018
ceph-iscsi-config-2.3-13.el7cp.noarch                       Sat Jun 16 02:08:35 2018
libcephfs2-12.2.4-10.el7cp.x86_64                           Sat Jun 16 02:07:57 2018
python-cephfs-12.2.4-10.el7cp.x86_64                        Sat Jun 16 02:07:57 2018

tcmu-runner-1.3.0-0.7.el7.x86_64                            Sat Jun 16 18:28:56 2018


How reproducible:

100%reproduced

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:

Fix this issue ASAP

Additional info:
Comment 7 Mike Christie 2018-09-11 15:44:38 UTC 
We have a fix or this in bz:

https://bugzilla.redhat.com/show_bug.cgi?id=1623737
Comment 8 Mike Christie 2018-09-12 15:29:09 UTC 

*** This bug has been marked as a duplicate of bug 1623737 ***