Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1443178

Summary: EMC VNX driver fails to detach volume / delete volume [multipath]
Product: Red Hat OpenStack Reporter: Jeremy <jmelvin>
Component: python-os-brickAssignee: Gorka Eguileor <geguileo>
Status: CLOSED ERRATA QA Contact: Avi Avraham <aavraham>
Severity: high Docs Contact:
Priority: high    
Version: 10.0 (Newton)CC: abeausol, apevec, eharney, geguileo, jjoyce, jmelvin, jschluet, lhh, pgrist, samccann, srevivo, tshefi
Target Milestone: z5Keywords: OtherQA, Triaged, Unconfirmed, ZStream
Target Release: 10.0 (Newton)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: python-os-brick-1.6.1-5.el7ost Doc Type: Bug Fix
Doc Text:
Complete refactoring of the iSCSI connection mechanism for improved reliability. For optimum results, use with openstack-cinder version 9.1.4-7 or higher and iscsi-initiator-utils version 6.2.0.874-2 or higher.
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-09-28 16:35:22 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jeremy 2017-04-18 17:01:27 UTC
Description of problem: I am able to create volumes with emc vnx driver and attach them to an instance. However when I attempt detach it stuck detaching , nothing in debug cinder volume or nova-compute logs. However when I manually detach the volome in the db and do cinder delete it still fails to delete with error:

2017-04-17 20:03:35.143 154877 INFO storops.vnx.navi_command [-] call command: /opt/Navisphere/bin/naviseccli -h 10.49.201.235 -user sysadmin -password *** -scope GLOBAL lun -destroy -name volume-89c6d587-b30d-4e04-b608-3a4ca1bdc545 -forceDetach -o
2017-04-17 20:03:36.757 154877 DEBUG storops.vnx.navi_command [-] command complete: /opt/Navisphere/bin/naviseccli -h 10.49.201.235 -user sysadmin -password *** -scope GLOBAL lun -destroy -name volume-89c6d587-b30d-4e04-b608-3a4ca1bdc545 -forceDetach -o, time consumed (s): 1.61393117905, output:
Cannot unbind LUN because it's contained in a Storage Group _log_output /usr/lib/python2.7/site-packages/storops/vnx/navi_command.py:190
2017-04-17 20:03:36.829 154877 ERROR oslo_messaging.rpc.server [req-9d4f0e8a-de24-4494-8b5c-8ef4e00b9adb 3ccfe3c6fb0a4f1bbaba1ab5c6d9d12e 37d5d10589c34022b7388d78fc7ad427 - default default] Exception during message handling
2017-04-17 20:03:36.829 154877 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/storops/exception.py", line 227, in raise_if_err
2017-04-17 20:03:36.829 154877 ERROR oslo_messaging.rpc.server     raise ex_clz(msg)
2017-04-17 20:03:36.829 154877 ERROR oslo_messaging.rpc.server VNXLunInStorageGroupError: failed to remove lun volume-89c6d587-b30d-4e04-b608-3a4ca1bdc545  detail:
2017-04-17 20:03:36.829 154877 ERROR oslo_messaging.rpc.server Cannot unbind LUN because it's contained in a Storage Group
2017-04-17 20:03:36.829 154877 ERROR oslo_messaging.rpc.server


Version-Release number of selected component (if applicable):
openstack-cinder	9.0.0-13.el7ost

How reproducible:
100%

Steps to Reproduce:
1.create emc vnx volume
2. attach
3. detach

Actual results:
fails detach / fails delete 

Expected results:
completes detach and delete 

Additional info:
Seems similar to : https://bugzilla.redhat.com/show_bug.cgi?id=1263010
, however this was solved in osp6 ?

Comment 1 Red Hat Bugzilla Rules Engine 2017-04-18 17:01:34 UTC
This bugzilla has been removed from the release and needs to be reviewed and Triaged for another Target Release.

Comment 3 Eric Harney 2017-04-18 17:46:40 UTC
The driver has an option "force_delete_lun_in_storagegroup" which may be related to this issue, but the description doesn't indicate why an admin would turn it on or off, or what the effects may be.

Comment 4 Jeremy 2017-04-18 19:17:33 UTC
VNX section of the Newton docs "https://docs.openstack.org/newton/config-reference/block-storage/drivers/emc-vnx-driver.html", and noticed this option:

Force delete volumes in storage group¶
Some available volumes may remain in storage group on the VNX array due to some OpenStack timeout issue. But the VNX array do not allow the user to delete the volumes which are in storage group. Option force_delete_lun_in_storagegroup is introduced to allow the user to delete the available volumes in this tricky situation.

When force_delete_lun_in_storagegroup is set to True in the back-end section, the driver will move the volumes out of the storage groups and then delete them if the user tries to delete the volumes that remain in the storage group on the VNX array.

The default value of force_delete_lun_in_storagegroup is False.


Customer added that option to the controller which is hosting the volume service, and now able to delete the volume that are stuck in 'error_deleting'.

Comment 5 Jeremy 2017-04-19 16:44:04 UTC
still having issues on other compute nodes:



###create volume with type emxvnx
###create vm on compute node
###attach volume (success)
2017-04-19 16:15:22.850 808426 INFO nova.compute.manager [req-4b0e6ca0-cae7-4402-afc9-9d62aa71c788 3ccfe3c6fb0a4f1bbaba1ab5c6d9d12e 37d5d10589c34022b7388d78fc7ad427 - - -] [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f] Attaching volume cadf43c2-fc9c-4581-9bc1-b0b22566d172 to /dev/vdb




###attempt detach volume (hangs)
2017-04-19 16:16:00.051 808426 INFO nova.compute.manager [req-6e964294-4e86-4ecc-a997-98f52ce41b07 3ccfe3c6fb0a4f1bbaba1ab5c6d9d12e 37d5d10589c34022b7388d78fc7ad427 - - -] [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f] Detach volume cadf43c2-fc9c-4581-9bc1-b0b22566d172 from mountpoint /dev/vdb

###restart openstack-nova-compute



###volume attach fails with this:

2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [req-6e964294-4e86-4ecc-a997-98f52ce41b07 3ccfe3c6fb0a4f1bbaba1ab5c6d9d12e 37d5d10589c34022b7388d78fc7ad427 - - -] [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f] Failed to detach volume cadf43c2-fc9c-4581-9bc1-b0b22566d172 from /dev/vdb
2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f] Traceback (most recent call last):
2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4719, in _driver_detach_volume
2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f]     encryption=encryption)
2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1298, in detach_volume
2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f]     self._disconnect_volume(connection_info, disk_dev)
2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1085, in _disconnect_volume
2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f]     vol_driver.disconnect_volume(connection_info, disk_dev)
2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume/iscsi.py", line 74, in disconnect_volume
2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f]     self.connector.disconnect_volume(connection_info['data'], None)
2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f]   File "/usr/lib/python2.7/site-packages/os_brick/utils.py", line 137, in trace_logging_wrapper
2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f]     return f(*args, **kwargs)
2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f]   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f]     return f(*args, **kwargs)
2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f]   File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py", line 492, in disconnect_volume
2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f]     connection_properties, multipath_device)
2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f]   File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py", line 649, in _disconnect_volume_multipath_iscsi
2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f]     self._disconnect_mpath(connection_properties, ips_iqns)
2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f]   File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py", line 774, in _disconnect_mpath
2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f]     self._rescan_multipath()
2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f]   File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py", line 844, in _rescan_multipath
2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f]     self._run_multipath(['-r'], check_exit_code=[0, 1, 21])
2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f]   File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py", line 830, in _run_multipath
2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f]     check_exit_code=check_exit_code)
2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f]   File "/usr/lib/python2.7/site-packages/os_brick/privileged/rootwrap.py", line 57, in execute
2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f]     return execute_root(*cmd, **kwargs)
2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f]   File "/usr/lib/python2.7/site-packages/oslo_privsep/priv_context.py", line 204, in _wrap
2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f]     return self.channel.remote_call(name, args, kwargs)
2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f]   File "/usr/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 177, in remote_call
2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f]     result = self.send_recv((Message.CALL.value, name, args, kwargs))
2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f]   File "/usr/lib/python2.7/site-packages/oslo_privsep/comm.py", line 159, in send_recv
2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f]     reply = future.result()
2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f]   File "/usr/lib/python2.7/site-packages/oslo_privsep/comm.py", line 105, in result
2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f]     raise self.error
2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f] IOError: Premature eof waiting for privileged process
2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f]
2017-04-19 16:21:03.018 808426 ERROR oslo_messaging.rpc.server [req-6e964294-4e86-4ecc-a997-98f52ce41b07 3ccfe3c6fb0a4f1bbaba1ab5c6d9d12e 37d5d10589c34022b7388d78fc7ad427 - - -] Exception during message handling


###reset volume state
###detach volume (works)

Comment 6 Eric Harney 2017-04-24 21:17:30 UTC
2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f]   File "/usr/lib/python2.7/site-packages/oslo_privsep/comm.py", line 105, in result
2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f]     raise self.error
2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f] IOError: Premature eof waiting for privileged process


This looks like a different issue -- oslo.privsep appears to be failing here.

Can you get an sosreport for that issue?

Comment 9 Jeremy 2017-04-25 19:00:44 UTC
Update:
There was a problem with the time on the controller and computes. Fixed that and it works. However when we try with multipath it fails.

[root@lvsnprod-compute-7 ~]# systemctl status multipathd
● multipathd.service - Device-Mapper Multipath Device Controller
   Loaded: loaded (/usr/lib/systemd/system/multipathd.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2017-04-25 18:26:42 UTC; 7min ago
  Process: 852517 ExecStart=/sbin/multipathd (code=exited, status=0/SUCCESS)
  Process: 852513 ExecStartPre=/sbin/multipath -A (code=exited, status=0/SUCCESS)
  Process: 852511 ExecStartPre=/sbin/modprobe dm-multipath (code=exited, status=0/SUCCESS)
 Main PID: 852520 (multipathd)
   CGroup: /system.slice/multipathd.service
           └─852520 /sbin/multipathd

Apr 25 18:32:45 lvsnprod-compute-7.osp.gspt.net multipathd[852520]: mpatho: load table [0 10485760 multipath 2 queue_if_no_path retain_attached_hw_handler 1 alua 1 1 round-rob... 8:192 1]
Apr 25 18:32:45 lvsnprod-compute-7.osp.gspt.net multipathd[852520]: sdj [8:144]: path removed from map mpatho
Apr 25 18:32:45 lvsnprod-compute-7.osp.gspt.net multipathd[852520]: sdm: remove path (uevent)
Apr 25 18:32:45 lvsnprod-compute-7.osp.gspt.net multipathd[852520]: mpatho: map in use
Apr 25 18:32:45 lvsnprod-compute-7.osp.gspt.net multipathd[852520]: mpatho: can't flush
Apr 25 18:32:45 lvsnprod-compute-7.osp.gspt.net multipathd[852520]: mpatho: load table [0 10485760 multipath 2 queue_if_no_path retain_attached_hw_handler 1 alua 0 0]
Apr 25 18:32:45 lvsnprod-compute-7.osp.gspt.net multipathd[852520]: mpatho: Entering recovery mode: max_retries=60
Apr 25 18:32:45 lvsnprod-compute-7.osp.gspt.net multipathd[852520]: sdm [8:192]: path removed from map mpatho
Apr 25 18:32:45 lvsnprod-compute-7.osp.gspt.net multipathd[852520]: mpatho: Entering recovery mode: max_retries=60
Apr 25 18:32:45 lvsnprod-compute-7.osp.gspt.net multipathd[852520]: mpatho: Entering recovery mode: max_retries=60
Hint: Some lines were ellipsized, use -l to show in full.
[root@lvsnprod-compute-7 ~]# multipath -ll
mpatho (360060160a3c03c003971fa46e529e711) dm-1
size=5.0G features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw
mpathn (360060160a3c03c003b71fa46e529e711) dm-0
size=5.0G features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw
[root@lvsnprod-compute-7 ~]#


I believe maptho and mpathn is the 5G vol I attached to the VM on that compute node.  Because this can't be detached, the delete of my heat stack is hanging.

And in the messages file, I see this:

pr 25 14:32:44 lvsnprod-compute-7 multipathd: mpathn: load table [0 10485760 multipath 2 queue_if_no_path retain_attached_hw_handler 1 alua 2 1 round-robin 0 1 1 8:32 1 round-robin 0 2 1 8:64 1 8:128 1]
Apr 25 14:32:44 lvsnprod-compute-7 multipathd: sdg [8:96]: path removed from map mpathn
Apr 25 14:32:44 lvsnprod-compute-7 multipathd: mpathn: load table [0 10485760 multipath 2 queue_if_no_path retain_attached_hw_handler 1 alua 1 1 round-robin 0 2 1 8:64 1 8:128 1]
Apr 25 14:32:44 lvsnprod-compute-7 multipathd: sdc [8:32]: path removed from map mpathn
Apr 25 14:32:44 lvsnprod-compute-7 multipathd: mpathn: load table [0 10485760 multipath 2 queue_if_no_path retain_attached_hw_handler 1 alua 1 1 round-robin 0 1 1 8:128 1]
Apr 25 14:32:44 lvsnprod-compute-7 multipathd: sde [8:64]: path removed from map mpathn
Apr 25 14:32:44 lvsnprod-compute-7 multipathd: mpathn: map in use
Apr 25 14:32:44 lvsnprod-compute-7 multipathd: mpathn: can't flush
Apr 25 14:32:44 lvsnprod-compute-7 multipathd: mpathn: load table [0 10485760 multipath 2 queue_if_no_path retain_attached_hw_handler 1 alua 0 0]
Apr 25 14:32:44 lvsnprod-compute-7 multipathd: mpathn: Entering recovery mode: max_retries=60
Apr 25 14:32:44 lvsnprod-compute-7 multipathd: sdi [8:128]: path removed from map mpathn
Apr 25 14:32:44 lvsnprod-compute-7 multipathd: mpathn: Entering recovery mode: max_retries=60
Apr 25 14:32:44 lvsnprod-compute-7 multipathd: mpathn: Entering recovery mode: max_retries=60
Apr 25 14:32:45 lvsnprod-compute-7 multipathd: mpatho: load table [0 10485760 multipath 2 queue_if_no_path retain_attached_hw_handler 1 alua 2 1 round-robin 0 1 1 8:176 1 round-robin 0 2 1 8:144 1 8:192 1]
Apr 25 14:32:45 lvsnprod-compute-7 multipathd: sdk [8:160]: path removed from map mpatho
Apr 25 14:32:45 lvsnprod-compute-7 multipathd: mpatho: load table [0 10485760 multipath 2 queue_if_no_path retain_attached_hw_handler 1 alua 1 1 round-robin 0 2 1 8:144 1 8:192 1]
Apr 25 14:32:45 lvsnprod-compute-7 multipathd: sdl [8:176]: path removed from map mpatho
Apr 25 14:32:45 lvsnprod-compute-7 multipathd: mpatho: load table [0 10485760 multipath 2 queue_if_no_path retain_attached_hw_handler 1 alua 1 1 round-robin 0 1 1 8:192 1]
Apr 25 14:32:45 lvsnprod-compute-7 multipathd: sdj [8:144]: path removed from map mpatho
Apr 25 14:32:45 lvsnprod-compute-7 multipathd: mpatho: map in use
Apr 25 14:32:45 lvsnprod-compute-7 multipathd: mpatho: can't flush
Apr 25 14:32:45 lvsnprod-compute-7 multipathd: mpatho: load table [0 10485760 multipath 2 queue_if_no_path retain_attached_hw_handler 1 alua 0 0]
Apr 25 14:32:45 lvsnprod-compute-7 multipathd: mpatho: Entering recovery mode: max_retries=60
Apr 25 14:32:45 lvsnprod-compute-7 multipathd: sdm [8:192]: path removed from map mpatho
Apr 25 14:32:45 lvsnprod-compute-7 multipathd: mpatho: Entering recovery mode: max_retries=60
Apr 25 14:32:45 lvsnprod-compute-7 multipathd: mpatho: Entering recovery mode: max_retries=60


So looks like this is a multipath issue.

Comment 10 Jeremy 2017-04-25 20:06:01 UTC
example of failed detach multipath issue:

[heat-admin@lvsnprod-controller-1 ~]$ cinder list
+--------------------------------------+-----------+---------------+------+-------------+----------+--------------------------------------+
| ID                                   | Status    | Name          | Size | Volume Type | Bootable | Attached to                          |
+--------------------------------------+-----------+---------------+------+-------------+----------+--------------------------------------+
| 0a69bf68-8f16-403c-a7df-18a66bf72b7f | detaching | vol-for-node1 | 5    | emcvnx01    | false    | 38c4426b-5da7-437d-9581-cc82e8e3851a |
+--------------------------------------+-----------+---------------+------+-------------+----------+--------------------------------------+
[heat-admin@lvsnprod-controller-1 ~]$ nova list
+--------------------------------------+--------+--------+------------+-------------+------------------------------+
| ID                                   | Name   | Status | Task State | Power State | Networks                     |
+--------------------------------------+--------+--------+------------+-------------+------------------------------+
| 38c4426b-5da7-437d-9581-cc82e8e3851a | node1  | ACTIVE | -          | Running     | net-10_146_99_0=10.146.99.41 |
| 6e49cfab-6257-4901-bcb3-2de44f30da77 | tonyvm | ACTIVE | -          | Running     | net-10_146_97_0=10.146.97.40 |
+--------------------------------------+--------+--------+------------+-------------+------------------------------+

Updated sosreports for the multipath issue.
 [extracting] sosreport-lvsnprod-controller-0.osp.gspt.net-20170425191151.tar.xz
 [extracting] sosreport-lvsnprod-compute-7.osp.gspt.net-20170425191135.tar.xz
 [extracting] sosreport-lvsnprod-controller-2.osp.gspt.net-20170425192449.tar.xz
 [parsing] sosreport for lvsnprod-compute-7.osp.gspt.net [7.3] -- Tue Apr 25 19:11:44 2017
 [parsing] sosreport for lvsnprod-controller-0.osp.gspt.net [7.3] -- Tue Apr 25 19:11:59 2017
 [parsing] sosreport for lvsnprod-controller-2.osp.gspt.net [7.3] -- Tue Apr 25 19:25:15 2017
 [scanning] for and moving additional files
 [erasing] empty directories
---
 [yank] complete - access files in /cases/01832247
 [browse] the files here: http://collab-shell.usersys.redhat.com/01832247/
please report any issues here: https://gitlab.cee.redhat.com/gss-tools/collab-shell/issues

Comment 11 Gorka Eguileor 2017-05-08 15:59:54 UTC
Looks like the reason why the volume is never detached is because in the compute node Nova calls OS-Bricks disconnect and in a call to `multipath -r` it hangs forever and never returns, so Nova doesn't call Cinder to complete the detach:

2017-04-25 18:32:45.277 818820 DEBUG oslo.privsep.daemon [-] privsep: request[149767024]: (3, 'os_brick.privileged.rootwrap.execute_root', ('multipath', '-r'), {'check_exit_code': (0, 1, 21)}) out_of_band /usr/lib/python2.7/site-packages/oslo_privsep/daemon.py:194
2017-04-25 18:32:45.277 818820 DEBUG oslo.privsep.daemon [-] Running cmd (subprocess): multipath -r out_of_band /usr/lib/python2.7/site-packages/oslo_privsep/daemon.py:194

Hanging there is unusual, but we are working on a complete refactoring of the iSCSI os-brick code and we won't be using the `multipath -r` anymore in that refactoring.


If you want to delete that volume you should change the status and attach-status with the `cinder reset-state` command to set it to available and detached and remove the mapping using the backend's management tool.

Comment 20 Tzach Shefi 2017-09-28 09:26:10 UTC
Setting OtherQA don't have VNX to test this any way. 

Also this happens with calls to multipath -r which we don't use any more.

Comment 22 errata-xmlrpc 2017-09-28 16:35:22 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2825