Bug 677821 - multipathd occassionally doesn't stop queuing after no_path_retry times out.
multipathd occassionally doesn't stop queuing after no_path_retry times out.
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: device-mapper-multipath (Show other bugs)
5.6
Unspecified Unspecified
unspecified Severity unspecified
: rc
: ---
Assigned To: Ben Marzinski
Gris Ge
: ZStream
Depends On:
Blocks: 683447
  Show dependency treegraph
 
Reported: 2011-02-15 18:27 EST by Ben Marzinski
Modified: 2011-07-21 04:23 EDT (History)
14 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
If a device's last path was deleted while the multipathd daemon was trying to reload the device map, or if a ghost path failed, multipathd did not always switch into the recovery mode. As a result, multipath devices could not recover I/O operations in setups that were supposed to temporarily queue I/O if all paths were down. Multipath now correctly recovers I/O operations as configured.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2011-07-21 04:23:12 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Ben Marzinski 2011-02-15 18:27:03 EST
There are two bugs in multipathd that keep it from switching into recovery mode when the last path has failed, and then disabling queue_if_no_path.

First off, during device deletion, if a multipath device tries to reload when the last path fails, but the path gets deleted while it is trying, the multipath device can lose track of what its device type is.  This keeps it from using the device specific configuration.  If the no_path_retry value in the defaults section doesn't match the value in the devices section for the multipath device, the device may fail to disable queueing.

Second, When calculating the number of active paths, multipath usually counts both active and ghost paths. However there is one place in the code where only active paths are counted.  If a ghost path fails after the number of active paths has been counted using only active paths, the number of active paths will be wrong.  Since multipath will only go into recovery mode when there are 0 active paths, this will keep multipath from going into recovery mode at the right time.
Comment 9 Ben Marzinski 2011-04-07 22:51:01 EDT
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
If a device's last path was deleted while the multipathd daemon was trying to reload the device map, or if a ghost path failed, multipathd did not always switch into the recovery mode. As a result, multipath devices could not recover I/O operations in setups that were supposed to temporarily queue I/O if all paths were down. Multipath now correctly recovers I/O operations as configured.
Comment 14 Ben Marzinski 2011-04-27 12:00:44 EDT
This is a rhel5 bug, and this fix is built into the rhel5 tarball. However, now that I look at RHEL6, there's a part of this fix that is missing.  The rest of it is also already in the rhel6 tarball. I'm going to open a bug for the missing piece.
Comment 16 Gris Ge 2011-05-30 01:33:10 EDT
for queue_if_no_path issue, it has been fixed by device-mapper-multipath-0.4.7-46.el5.

Tested with emc_clariion_checker and set up no_path_retry as 5 in devices setcion of multipath.conf

Using dd for generating I/O. And these command for bring disk offline:
===
for X in `echo "sdg sdi sdw sdy"`;do
    echo offline > "/sys/block/$X/device/state"
done
===
Before patch, we will got incorrect no_path_retry_nr:
===
mpath5: Entering recovery mode: max_retries=60
===
After:
mpath5: Entering recovery mode: max_retries=5


For PATH_GHOST issue, not able to reproduce it as lacking of specified path_checker: "rdac, tur, and hp_sw path checkers"

Also tried to change path_checker of DGC, but still not able to reproduce it.

Can we get any partner test to PATH_GHOST issue?
Comment 22 Gris Ge 2011-06-23 05:21:26 EDT
Tried 100 times on EMC CX (active/passive setup) using  tur path_checker.

Cannot hit active paths: <negative_number> issue.

Basic function test will be noted in errata.

Sanity Only.
Comment 23 errata-xmlrpc 2011-07-21 04:23:12 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-1032.html

Note You need to log in before you can comment on or make changes to this bug.