Bug 1255523 - Controller node does not fully detach multipath device and the device can not be removed by manual means
Controller node does not fully detach multipath device and the device can not...
Status: CLOSED ERRATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-cinder (Show other bugs)
5.0 (RHEL 6)
Unspecified Unspecified
urgent Severity urgent
: async
: 5.0 (RHEL 6)
Assigned To: Gorka Eguileor
nlevinki
: ZStream
Depends On:
Blocks: 1280401 1280409 1278590
  Show dependency treegraph
 
Reported: 2015-08-20 16:05 EDT by Dave Sullivan
Modified: 2016-04-26 11:09 EDT (History)
9 users (show)

See Also:
Fixed In Version: openstack-cinder-2014.1.5-2.el6ost
Doc Type: Bug Fix
Doc Text:
Previously, when systems were under heavy load, multipath detection would take longer than expected and the device would not be detached if the validation failed. As a result, Block Storage did not detect the existence of the multipath in time and would work in the single path mode even when the underlying operating system detected the multipath. When the device was disconnected, it was as if it was a single path device, and end up with leftovers in the system. On the multipath, even though the paths were properly detected, they were all in a failed state when tried to read from the device. With this update, Block Storage is configured for multipath and there are several retries to detect multipaths on connect. Also there are checks on disconnect if a device that was single path changes to a multipath. As a result, there are no leftovers devices attaches on validation failures.
Story Points: ---
Clone Of:
: 1278590 1280401 1280409 (view as bug list)
Environment:
Last Closed: 2015-12-22 09:26:20 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Dave Sullivan 2015-08-20 16:05:02 EDT
Description of problem:

Load testing of the following commands eventually finds a situation where the controller node shows a faulting mpio device that also shows as a backed mpio device for a launched instance on the compute node.

lsof | grep dm-12 #for the faulting mpio path

showed blkid and kpartx

low levels attempts were made to kill these off first with kill -15 (workded  for blkid) but kill -9 needed on kpartx

were still not able to delete the faulty path

This is using rhos5 on rhel6 using hp 3par backend.




Version-Release number of selected component (if applicable):




How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:


Proper (online) cleanup procedure for failed path (multipath -F is intrusive!!!):

======

MPATHDEV="/dev/dm-58"
multipath -ll $MPATHDEV

for i in $( multipath -ll $MPATHDEV | awk '/ failed / { print $3 }' 
do 
  echo "Removing: $i"; echo 1 > /sys/block/${i}/device/delete
done

multipath -ll $MPATHDEV
multipath -f $MPATHDEV

=====

This seems like a different issue but worth noting here.

multipath errors being parsed as device names.
https://bugzilla.redhat.com/show_bug.cgi?id=1235786
Comment 4 Dave Sullivan 2015-08-20 16:43:12 EDT
mushroom configured to do this eventually runs into this mpio problem

==================================================================
step 1: create a volume via cinder

cinder create --image-id 32144d4b-cfaf-4038-8fb8-feb037d8723d --volume-type LocalLVM --display-name test1001 40

step 2:

nova --debug boot vm-test1001 --flavor 3 --block-device source=volume,id=da49ff15-f914-4bf3-8a8c-b3cc3f27d734,dest=volume,size=40,shutdown=remove,bootindex=0
=================================================================
Comment 5 Dave Sullivan 2015-08-20 16:52:05 EDT
Customer seems to indicate this problem shows up quicker from Case 1 approach.

(1)

1.1 nova create volume

1.2 nova boot from volume (has delete_on_terminate)

 

nova delete instance

cinder delete volume

..... faulty multipath hangs around (it is gone after a few minutes)

 

(2)

2.1 cinder create volume

2.2 nova boot from volume (does not have delete_on_terminate)

 

nova delete instance

cinder delete volume

,,,,, no fauly multipath hangs around

 

 

nova volume-create 40 --volume-type=premium --image-id=2302496c-7384-495d-96bc-03a26a28dd41 --display-name=just-premium-vol-606

 

to

 

cinder create --image-id 2302496c-7384-495d-96bc-03a26a28dd41 --volume-type premium --display-name test707 40
Comment 6 Dave Sullivan 2015-08-20 17:22:25 EDT
other notes, seems like a race/load type condition, it requires mushroom driven load of comment #5
Comment 8 Sergey Gotliv 2015-08-25 05:23:56 EDT
It looks like Cinder configuration issue. Please, set

use_multipath_for_image_xfer = true

in /etc/cinder.conf.

Today it's commented out therefore Cinder is using a default value, which is false. In that case, Cinder is removing a single path instead of the multipath
device. Later today we have a call with the customer to confirm that theory.
Comment 9 John Fulton 2015-08-25 12:20:49 EDT
After adding the following to /etc/cinder/cinder.conf

[HP3PARFC]
use_multipath_for_image_xfer=true

As opposed to just globally adding use_multipath_for_image_xfer=true we
no longer saw faulty paths. The customer is going to run the test at
scale and let me know the results. 

Should I open a separate bug to address the need to have to do this per device? An upstream bug already seems to exist: 

https://bugs.launchpad.net/cinder/+bug/1326571
Comment 10 John Fulton 2015-08-26 17:43:00 EDT
The issue is much improved after the xfer setting in comment 9, but after testing it has recurred. The customer did a non-concurrent test like the following: 

for x in range(0, 128):
    vol = cinder_create()
    vm = nova_boot(vol)
    nova_delete(vm)
    # cinder_delete is implicit because of nova boot's shutdown=remove option

After 128 loops we saw one faulty path remaining. I will follow up with more details.
Comment 12 Sergey Gotliv 2015-09-01 08:20:20 EDT
Cinder makes a decision to delete a single path or the multipath device base on the multipath_id which is presumably created during attach_volume process.
After reviewing logs from the last debugging session I see that in many cases we trying to delete a single path because the multipath_id is missing. We still have to understand whether it's missing because the multipath device doesn't exist then it makes sense, but if it's missing in Cinder metadata only then it can explain why after detach Cinder leaves a faulty devices behind. We need to run another debug session.
Comment 13 John Fulton 2015-09-25 01:16:20 EDT
The reproducer test was modified as follows: 

- create a volume from a glance image (nova is not involved)
- four threads of creating two volumes as described above
- after 8 volumes are created, delete them
- using a different 3PAR array 
- using the same OpenStack install (3 controllers in HA + N compute)

The results were one faulty path as per `multipath -ll`.
Comment 18 Sergey Gotliv 2015-10-19 03:08:09 EDT
It seems that the issue is gone after latest multipath reconfgiuration.
Comment 21 Sergey Gotliv 2015-11-09 05:41:18 EST
Not all patches are merged, moving this bug back to POST.
Comment 24 errata-xmlrpc 2015-12-22 09:26:20 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2686.html

Note You need to log in before you can comment on or make changes to this bug.