Bug 474855 - Unexpected LUN failover during system restart
Unexpected LUN failover during system restart
Status: CLOSED DUPLICATE of bug 500998
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: device-mapper-multipath (Show other bugs)
5.2
x86_64 Linux
low Severity high
: rc
: ---
Assigned To: Ben Marzinski
Cluster QE
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2008-12-05 11:43 EST by gerbrecht
Modified: 2010-05-12 13:29 EDT (History)
16 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-05-12 13:29:14 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
IBM Linux Technology Center 48680 None None None Never

  None (edit)
Description gerbrecht 2008-12-05 11:43:49 EST
Description of problem:

We are running Red Hat Enterprise Linux 5.2 with Device-Mapper Multipath (device-mapper-multipath-0.4.7-17.el5) support.

The attached storage unit is a dual controller active/passive device (LSI/Engenio DS3x00/DS4x00). This storage device requires a dm_rdac hardware handler kernel module. Each node has 2 HBAs. This results into 4 paths from a node to each LUN. There are about 70 LUNs. For normal operation 2 paths should be in active/ready (preferred controller) state while the remaining 2 paths are in active/ghost (alternate controller) state.

The problem we see is when rebooting any node of the associated SoFS Cluster, some disks will failover to their alternate (non-preferred) path. Checking the storage unit log proves that dm_rdac has issued mode select commands causing logical drives to leave their preferred path due to RDAC failover.

One root cause seems to be that LUN enumeration hasn't finished when the multipathd daemon is started.

This causes several problems:

The path information from 'multipath -ll' and 'show topology' (interactive interface mode 'multipathd -k') is inconsistent. In case of around 70 LUNs adding 'sleep 300' to the start() section of the multipathd init script prior to the daemon start will take care of this.

While the system is started up a range of blacklisted devices can be watched moving down through the device list. These seem to be the most recently discovered devices. The display is done by 'show devices'. Very often while discovery moves on some devices stay blacklisted for ever.

This causes a non deterministic number of devices to be blacklisted despite never being added to any black list. A restart of the multipathd daemon or 'reconfigure' in interactive mode after all LUNs have been enumerated will cleanup everything.

The RDAC failover is a side effect if the blacklisted devices happen to be members of preferred paths. This causes massive problems for all other nodes in our cluster. 

Version-Release number of selected component (if applicable):
Red Hat Enterprise Linux 5.2
device-mapper-multipath-0.4.7-17.el5)

How reproducible:
There needs to be a large number of LUNs and an active/passive storage unit.
This will produce lot's of I/O errors during startup because will be read access to ghost paths. The startup is delayed by those I/O errors.

Actual results:
There are failovers of LUNs during startup.

Expected results:
There shouldn't be any failovers during startup if there is no point of failure.

Additional info:
/etc/multipath.conf

defaults {
        udev_dir                /dev
        getuid_callout          "/sbin/scsi_id -g -u -s /block/%n"
        user_friendly_names     yes
}

blacklist {
        devnode                 "^sda[0-9]*$"
}

devices {
        device {
                vendor                  "IBM"
                product                 "1726"
                failback                immediate
                no_path_retry           fail
                hardware_handler        "1 rdac"
                path_checker            rdac
                path_grouping_policy    group_by_prio
                prio_callout            "/sbin/mpath_prio_rdac /dev/%n"
        }

        device {
                vendor                  "IBM"
                product                 "1814"
                failback                immediate
                no_path_retry           fail
                hardware_handler        "1 rdac"
                path_checker            rdac
                path_grouping_policy    group_by_prio
                prio_callout            "/sbin/mpath_prio_rdac /dev/%n"
        }

        device {
                vendor                  "IBM"
                product                 "1815"
                failback                immediate
                no_path_retry           fail
                hardware_handler        "1 rdac"
                path_checker            rdac
                path_grouping_policy    group_by_prio
                prio_callout            "/sbin/mpath_prio_rdac /dev/%n"
        }
}
Comment 1 gerbrecht 2009-01-08 05:27:30 EST
Do you need additional information?
Comment 3 Ben Marzinski 2010-04-27 13:40:49 EDT
Sorry that this one slipped through the cracks somehow. It's a known issue that if multipathd starts up before all of your path devices have been setup, it will work with what it has. If it only sees a passive path to a device, it will failover to that passive path.  When the other paths come online, multipath should fail back to them.  I realize that this causes a lot of annoyance.  One solution is to make add your device driver to your initrd, so that the devices are discovered early in the boot sequence.
Comment 4 Ben Marzinski 2010-05-12 13:29:14 EDT

*** This bug has been marked as a duplicate of bug 500998 ***

Note You need to log in before you can comment on or make changes to this bug.