Bug 1443178
| Summary: | EMC VNX driver fails to detach volume / delete volume [multipath] | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Jeremy <jmelvin> |
| Component: | python-os-brick | Assignee: | Gorka Eguileor <geguileo> |
| Status: | CLOSED ERRATA | QA Contact: | Avi Avraham <aavraham> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 10.0 (Newton) | CC: | abeausol, apevec, eharney, geguileo, jjoyce, jmelvin, jschluet, lhh, pgrist, samccann, srevivo, tshefi |
| Target Milestone: | z5 | Keywords: | OtherQA, Triaged, Unconfirmed, ZStream |
| Target Release: | 10.0 (Newton) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | python-os-brick-1.6.1-5.el7ost | Doc Type: | Bug Fix |
| Doc Text: |
Complete refactoring of the iSCSI connection mechanism for improved reliability.
For optimum results, use with openstack-cinder version 9.1.4-7 or higher and iscsi-initiator-utils version 6.2.0.874-2 or higher.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-09-28 16:35:22 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Jeremy
2017-04-18 17:01:27 UTC
This bugzilla has been removed from the release and needs to be reviewed and Triaged for another Target Release. The driver has an option "force_delete_lun_in_storagegroup" which may be related to this issue, but the description doesn't indicate why an admin would turn it on or off, or what the effects may be. VNX section of the Newton docs "https://docs.openstack.org/newton/config-reference/block-storage/drivers/emc-vnx-driver.html", and noticed this option: Force delete volumes in storage group¶ Some available volumes may remain in storage group on the VNX array due to some OpenStack timeout issue. But the VNX array do not allow the user to delete the volumes which are in storage group. Option force_delete_lun_in_storagegroup is introduced to allow the user to delete the available volumes in this tricky situation. When force_delete_lun_in_storagegroup is set to True in the back-end section, the driver will move the volumes out of the storage groups and then delete them if the user tries to delete the volumes that remain in the storage group on the VNX array. The default value of force_delete_lun_in_storagegroup is False. Customer added that option to the controller which is hosting the volume service, and now able to delete the volume that are stuck in 'error_deleting'. still having issues on other compute nodes: ###create volume with type emxvnx ###create vm on compute node ###attach volume (success) 2017-04-19 16:15:22.850 808426 INFO nova.compute.manager [req-4b0e6ca0-cae7-4402-afc9-9d62aa71c788 3ccfe3c6fb0a4f1bbaba1ab5c6d9d12e 37d5d10589c34022b7388d78fc7ad427 - - -] [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f] Attaching volume cadf43c2-fc9c-4581-9bc1-b0b22566d172 to /dev/vdb ###attempt detach volume (hangs) 2017-04-19 16:16:00.051 808426 INFO nova.compute.manager [req-6e964294-4e86-4ecc-a997-98f52ce41b07 3ccfe3c6fb0a4f1bbaba1ab5c6d9d12e 37d5d10589c34022b7388d78fc7ad427 - - -] [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f] Detach volume cadf43c2-fc9c-4581-9bc1-b0b22566d172 from mountpoint /dev/vdb ###restart openstack-nova-compute ###volume attach fails with this: 2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [req-6e964294-4e86-4ecc-a997-98f52ce41b07 3ccfe3c6fb0a4f1bbaba1ab5c6d9d12e 37d5d10589c34022b7388d78fc7ad427 - - -] [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f] Failed to detach volume cadf43c2-fc9c-4581-9bc1-b0b22566d172 from /dev/vdb 2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f] Traceback (most recent call last): 2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4719, in _driver_detach_volume 2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f] encryption=encryption) 2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1298, in detach_volume 2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f] self._disconnect_volume(connection_info, disk_dev) 2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1085, in _disconnect_volume 2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f] vol_driver.disconnect_volume(connection_info, disk_dev) 2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume/iscsi.py", line 74, in disconnect_volume 2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f] self.connector.disconnect_volume(connection_info['data'], None) 2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f] File "/usr/lib/python2.7/site-packages/os_brick/utils.py", line 137, in trace_logging_wrapper 2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f] return f(*args, **kwargs) 2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f] File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner 2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f] return f(*args, **kwargs) 2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f] File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py", line 492, in disconnect_volume 2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f] connection_properties, multipath_device) 2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f] File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py", line 649, in _disconnect_volume_multipath_iscsi 2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f] self._disconnect_mpath(connection_properties, ips_iqns) 2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f] File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py", line 774, in _disconnect_mpath 2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f] self._rescan_multipath() 2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f] File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py", line 844, in _rescan_multipath 2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f] self._run_multipath(['-r'], check_exit_code=[0, 1, 21]) 2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f] File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py", line 830, in _run_multipath 2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f] check_exit_code=check_exit_code) 2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f] File "/usr/lib/python2.7/site-packages/os_brick/privileged/rootwrap.py", line 57, in execute 2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f] return execute_root(*cmd, **kwargs) 2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f] File "/usr/lib/python2.7/site-packages/oslo_privsep/priv_context.py", line 204, in _wrap 2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f] return self.channel.remote_call(name, args, kwargs) 2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f] File "/usr/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 177, in remote_call 2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f] result = self.send_recv((Message.CALL.value, name, args, kwargs)) 2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f] File "/usr/lib/python2.7/site-packages/oslo_privsep/comm.py", line 159, in send_recv 2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f] reply = future.result() 2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f] File "/usr/lib/python2.7/site-packages/oslo_privsep/comm.py", line 105, in result 2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f] raise self.error 2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f] IOError: Premature eof waiting for privileged process 2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f] 2017-04-19 16:21:03.018 808426 ERROR oslo_messaging.rpc.server [req-6e964294-4e86-4ecc-a997-98f52ce41b07 3ccfe3c6fb0a4f1bbaba1ab5c6d9d12e 37d5d10589c34022b7388d78fc7ad427 - - -] Exception during message handling ###reset volume state ###detach volume (works) 2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f] File "/usr/lib/python2.7/site-packages/oslo_privsep/comm.py", line 105, in result 2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f] raise self.error 2017-04-19 16:21:02.798 808426 ERROR nova.compute.manager [instance: 2f7cc0d3-500d-40bc-98f5-c68a60bbf35f] IOError: Premature eof waiting for privileged process This looks like a different issue -- oslo.privsep appears to be failing here. Can you get an sosreport for that issue?
Update:
There was a problem with the time on the controller and computes. Fixed that and it works. However when we try with multipath it fails.
[root@lvsnprod-compute-7 ~]# systemctl status multipathd
● multipathd.service - Device-Mapper Multipath Device Controller
Loaded: loaded (/usr/lib/systemd/system/multipathd.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2017-04-25 18:26:42 UTC; 7min ago
Process: 852517 ExecStart=/sbin/multipathd (code=exited, status=0/SUCCESS)
Process: 852513 ExecStartPre=/sbin/multipath -A (code=exited, status=0/SUCCESS)
Process: 852511 ExecStartPre=/sbin/modprobe dm-multipath (code=exited, status=0/SUCCESS)
Main PID: 852520 (multipathd)
CGroup: /system.slice/multipathd.service
└─852520 /sbin/multipathd
Apr 25 18:32:45 lvsnprod-compute-7.osp.gspt.net multipathd[852520]: mpatho: load table [0 10485760 multipath 2 queue_if_no_path retain_attached_hw_handler 1 alua 1 1 round-rob... 8:192 1]
Apr 25 18:32:45 lvsnprod-compute-7.osp.gspt.net multipathd[852520]: sdj [8:144]: path removed from map mpatho
Apr 25 18:32:45 lvsnprod-compute-7.osp.gspt.net multipathd[852520]: sdm: remove path (uevent)
Apr 25 18:32:45 lvsnprod-compute-7.osp.gspt.net multipathd[852520]: mpatho: map in use
Apr 25 18:32:45 lvsnprod-compute-7.osp.gspt.net multipathd[852520]: mpatho: can't flush
Apr 25 18:32:45 lvsnprod-compute-7.osp.gspt.net multipathd[852520]: mpatho: load table [0 10485760 multipath 2 queue_if_no_path retain_attached_hw_handler 1 alua 0 0]
Apr 25 18:32:45 lvsnprod-compute-7.osp.gspt.net multipathd[852520]: mpatho: Entering recovery mode: max_retries=60
Apr 25 18:32:45 lvsnprod-compute-7.osp.gspt.net multipathd[852520]: sdm [8:192]: path removed from map mpatho
Apr 25 18:32:45 lvsnprod-compute-7.osp.gspt.net multipathd[852520]: mpatho: Entering recovery mode: max_retries=60
Apr 25 18:32:45 lvsnprod-compute-7.osp.gspt.net multipathd[852520]: mpatho: Entering recovery mode: max_retries=60
Hint: Some lines were ellipsized, use -l to show in full.
[root@lvsnprod-compute-7 ~]# multipath -ll
mpatho (360060160a3c03c003971fa46e529e711) dm-1
size=5.0G features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw
mpathn (360060160a3c03c003b71fa46e529e711) dm-0
size=5.0G features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw
[root@lvsnprod-compute-7 ~]#
I believe maptho and mpathn is the 5G vol I attached to the VM on that compute node. Because this can't be detached, the delete of my heat stack is hanging.
And in the messages file, I see this:
pr 25 14:32:44 lvsnprod-compute-7 multipathd: mpathn: load table [0 10485760 multipath 2 queue_if_no_path retain_attached_hw_handler 1 alua 2 1 round-robin 0 1 1 8:32 1 round-robin 0 2 1 8:64 1 8:128 1]
Apr 25 14:32:44 lvsnprod-compute-7 multipathd: sdg [8:96]: path removed from map mpathn
Apr 25 14:32:44 lvsnprod-compute-7 multipathd: mpathn: load table [0 10485760 multipath 2 queue_if_no_path retain_attached_hw_handler 1 alua 1 1 round-robin 0 2 1 8:64 1 8:128 1]
Apr 25 14:32:44 lvsnprod-compute-7 multipathd: sdc [8:32]: path removed from map mpathn
Apr 25 14:32:44 lvsnprod-compute-7 multipathd: mpathn: load table [0 10485760 multipath 2 queue_if_no_path retain_attached_hw_handler 1 alua 1 1 round-robin 0 1 1 8:128 1]
Apr 25 14:32:44 lvsnprod-compute-7 multipathd: sde [8:64]: path removed from map mpathn
Apr 25 14:32:44 lvsnprod-compute-7 multipathd: mpathn: map in use
Apr 25 14:32:44 lvsnprod-compute-7 multipathd: mpathn: can't flush
Apr 25 14:32:44 lvsnprod-compute-7 multipathd: mpathn: load table [0 10485760 multipath 2 queue_if_no_path retain_attached_hw_handler 1 alua 0 0]
Apr 25 14:32:44 lvsnprod-compute-7 multipathd: mpathn: Entering recovery mode: max_retries=60
Apr 25 14:32:44 lvsnprod-compute-7 multipathd: sdi [8:128]: path removed from map mpathn
Apr 25 14:32:44 lvsnprod-compute-7 multipathd: mpathn: Entering recovery mode: max_retries=60
Apr 25 14:32:44 lvsnprod-compute-7 multipathd: mpathn: Entering recovery mode: max_retries=60
Apr 25 14:32:45 lvsnprod-compute-7 multipathd: mpatho: load table [0 10485760 multipath 2 queue_if_no_path retain_attached_hw_handler 1 alua 2 1 round-robin 0 1 1 8:176 1 round-robin 0 2 1 8:144 1 8:192 1]
Apr 25 14:32:45 lvsnprod-compute-7 multipathd: sdk [8:160]: path removed from map mpatho
Apr 25 14:32:45 lvsnprod-compute-7 multipathd: mpatho: load table [0 10485760 multipath 2 queue_if_no_path retain_attached_hw_handler 1 alua 1 1 round-robin 0 2 1 8:144 1 8:192 1]
Apr 25 14:32:45 lvsnprod-compute-7 multipathd: sdl [8:176]: path removed from map mpatho
Apr 25 14:32:45 lvsnprod-compute-7 multipathd: mpatho: load table [0 10485760 multipath 2 queue_if_no_path retain_attached_hw_handler 1 alua 1 1 round-robin 0 1 1 8:192 1]
Apr 25 14:32:45 lvsnprod-compute-7 multipathd: sdj [8:144]: path removed from map mpatho
Apr 25 14:32:45 lvsnprod-compute-7 multipathd: mpatho: map in use
Apr 25 14:32:45 lvsnprod-compute-7 multipathd: mpatho: can't flush
Apr 25 14:32:45 lvsnprod-compute-7 multipathd: mpatho: load table [0 10485760 multipath 2 queue_if_no_path retain_attached_hw_handler 1 alua 0 0]
Apr 25 14:32:45 lvsnprod-compute-7 multipathd: mpatho: Entering recovery mode: max_retries=60
Apr 25 14:32:45 lvsnprod-compute-7 multipathd: sdm [8:192]: path removed from map mpatho
Apr 25 14:32:45 lvsnprod-compute-7 multipathd: mpatho: Entering recovery mode: max_retries=60
Apr 25 14:32:45 lvsnprod-compute-7 multipathd: mpatho: Entering recovery mode: max_retries=60
So looks like this is a multipath issue.
example of failed detach multipath issue: [heat-admin@lvsnprod-controller-1 ~]$ cinder list +--------------------------------------+-----------+---------------+------+-------------+----------+--------------------------------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+---------------+------+-------------+----------+--------------------------------------+ | 0a69bf68-8f16-403c-a7df-18a66bf72b7f | detaching | vol-for-node1 | 5 | emcvnx01 | false | 38c4426b-5da7-437d-9581-cc82e8e3851a | +--------------------------------------+-----------+---------------+------+-------------+----------+--------------------------------------+ [heat-admin@lvsnprod-controller-1 ~]$ nova list +--------------------------------------+--------+--------+------------+-------------+------------------------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+--------+--------+------------+-------------+------------------------------+ | 38c4426b-5da7-437d-9581-cc82e8e3851a | node1 | ACTIVE | - | Running | net-10_146_99_0=10.146.99.41 | | 6e49cfab-6257-4901-bcb3-2de44f30da77 | tonyvm | ACTIVE | - | Running | net-10_146_97_0=10.146.97.40 | +--------------------------------------+--------+--------+------------+-------------+------------------------------+ Updated sosreports for the multipath issue. [extracting] sosreport-lvsnprod-controller-0.osp.gspt.net-20170425191151.tar.xz [extracting] sosreport-lvsnprod-compute-7.osp.gspt.net-20170425191135.tar.xz [extracting] sosreport-lvsnprod-controller-2.osp.gspt.net-20170425192449.tar.xz [parsing] sosreport for lvsnprod-compute-7.osp.gspt.net [7.3] -- Tue Apr 25 19:11:44 2017 [parsing] sosreport for lvsnprod-controller-0.osp.gspt.net [7.3] -- Tue Apr 25 19:11:59 2017 [parsing] sosreport for lvsnprod-controller-2.osp.gspt.net [7.3] -- Tue Apr 25 19:25:15 2017 [scanning] for and moving additional files [erasing] empty directories --- [yank] complete - access files in /cases/01832247 [browse] the files here: http://collab-shell.usersys.redhat.com/01832247/ please report any issues here: https://gitlab.cee.redhat.com/gss-tools/collab-shell/issues Looks like the reason why the volume is never detached is because in the compute node Nova calls OS-Bricks disconnect and in a call to `multipath -r` it hangs forever and never returns, so Nova doesn't call Cinder to complete the detach:
2017-04-25 18:32:45.277 818820 DEBUG oslo.privsep.daemon [-] privsep: request[149767024]: (3, 'os_brick.privileged.rootwrap.execute_root', ('multipath', '-r'), {'check_exit_code': (0, 1, 21)}) out_of_band /usr/lib/python2.7/site-packages/oslo_privsep/daemon.py:194
2017-04-25 18:32:45.277 818820 DEBUG oslo.privsep.daemon [-] Running cmd (subprocess): multipath -r out_of_band /usr/lib/python2.7/site-packages/oslo_privsep/daemon.py:194
Hanging there is unusual, but we are working on a complete refactoring of the iSCSI os-brick code and we won't be using the `multipath -r` anymore in that refactoring.
If you want to delete that volume you should change the status and attach-status with the `cinder reset-state` command to set it to available and detached and remove the mapping using the backend's management tool.
Setting OtherQA don't have VNX to test this any way. Also this happens with calls to multipath -r which we don't use any more. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:2825 |