Bug 481239

+++ This bug was initially created as a clone of Bug #461122 +++

Escalated to Bugzilla from IssueTracker

--- Additional comment from tao on 2008-09-04 08:22:37 EDT ---

Description of problem:

A server has access to 8 LUN:s from a Clariion SAN. Two of the LUN:s are private and the rest is shared and to be used in an Oracle RAC environment. Customer wants to make sure that the two private LUN:s always gets device name /dev/mpath/mpath0 and /dev/mpath/mpath1. We have configured multipath to use aliases to solve this. The shared LUN wwid:s are not specified in the multipath.conf file since device names are not important for these LUN:s (customer will use Oracle RAC filesystem on these LUN:s).

During boot all LUN:s are detected and the two private LUN:s are given the correct mpath device names. 

Output of multipath -ll below:

mpath1 (36006016023e0120097c1f6b47807dc11)
[size=17 GB][features="1 queue_if_no_path"][hwhandler="1 emc"]
\\_ round-robin 0 [prio=2][active]
 \\_ 2:0:1:1 sdaa 65:160 [active][ready]
 \\_ 1:0:1:1 sdk  8:160  [active][ready]
\\_ round-robin 0 [enabled]
 \\_ 2:0:0:1 sds  65:32  [active][ready]
 \\_ 1:0:0:1 sdc  8:32   [active][ready]

mpath0 (36006016023e01200ed7976c4717ddc11)
[size=17 GB][features="1 queue_if_no_path"][hwhandler="1 emc"]
\\_ round-robin 0 [prio=2][active]
 \\_ 2:0:0:0 sdr  65:16  [active][ready]
 \\_ 1:0:0:0 sdb  8:16   [active][ready]
\\_ round-robin 0 [enabled]
 \\_ 1:0:1:0 sdj  8:144  [active][ready]
 \\_ 2:0:1:0 sdz  65:144 [active][ready]

mpath9 (36006016023e012004c38ee6f737ddc11)
[size=34 GB][features="1 queue_if_no_path"][hwhandler="1 emc"]
\\_ round-robin 0 [prio=2][active]
 \\_ 2:0:1:7 sdag 66:0   [active][ready]
 \\_ 1:0:1:7 sdq  65:0   [active][ready]
\\_ round-robin 0 [enabled]
 \\_ 2:0:0:7 sdy  65:128 [active][ready]
 \\_ 1:0:0:7 sdi  8:128  [active][ready]

mpath8 (36006016023e012004a38ee6f737ddc11)
[size=34 GB][features="1 queue_if_no_path"][hwhandler="1 emc"]
\\_ round-robin 0 [prio=2][active]
 \\_ 2:0:1:6 sdaf 65:240 [active][ready]
 \\_ 1:0:1:6 sdp  8:240  [active][ready]
\\_ round-robin 0 [enabled]
 \\_ 2:0:0:6 sdx  65:112 [active][ready]
 \\_ 1:0:0:6 sdh  8:112  [active][ready]

mpath7 (36006016023e01200627384e5727ddc11)
[size=34 GB][features="1 queue_if_no_path"][hwhandler="1 emc"]
\\_ round-robin 0 [prio=2][active]
 \\_ 2:0:1:5 sdae 65:224 [active][ready]
 \\_ 1:0:1:5 sdo  8:224  [active][ready]
\\_ round-robin 0 [enabled]
 \\_ 2:0:0:5 sdw  65:96  [active][ready]
 \\_ 1:0:0:5 sdg  8:96   [active][ready]

mpath6 (36006016023e01200accd89df727ddc11)
[size=34 GB][features="1 queue_if_no_path"][hwhandler="1 emc"]
\\_ round-robin 0 [prio=2][active]
 \\_ 2:0:1:4 sdad 65:208 [active][ready]
 \\_ 1:0:1:4 sdn  8:208  [active][ready]
\\_ round-robin 0 [enabled]
 \\_ 2:0:0:4 sdv  65:80  [active][ready]
 \\_ 1:0:0:4 sdf  8:80   [active][ready]

mpath5 (36006016023e01200aacd89df727ddc11)
[size=34 GB][features="1 queue_if_no_path"][hwhandler="1 emc"]
\\_ round-robin 0 [prio=2][active]
 \\_ 2:0:1:3 sdac 65:192 [active][ready]
 \\_ 1:0:1:3 sdm  8:192  [active][ready]
\\_ round-robin 0 [enabled]
 \\_ 2:0:0:3 sdu  65:64  [active][ready]
 \\_ 1:0:0:3 sde  8:64   [active][ready]

mpath4 (36006016023e01200ed0aafd8727ddc11)
[size=34 GB][features="1 queue_if_no_path"][hwhandler="1 emc"]
\\_ round-robin 0 [prio=2][active]
 \\_ 2:0:1:2 sdab 65:176 [active][ready]
 \\_ 1:0:1:2 sdl  8:176  [active][ready]
\\_ round-robin 0 [enabled]
 \\_ 2:0:0:2 sdt  65:48  [active][ready]
 \\_ 1:0:0:2 sdd  8:48   [active][ready]



If, for some reason, we want to rebuild the multipath config we do this:

# multipath -F

To recreate the config:

# multipath -v2

The output looks like this:

create: mpath1 (36006016023e0120097c1f6b47807dc11)
[size=17 GB][features="1 queue_if_no_path"][hwhandler="1 emc"]
\\_ round-robin 0 [prio=2]
 \\_ 2:0:1:1 sdaa 65:160 [ready]
 \\_ 1:0:1:1 sdk  8:160  [ready]
\\_ round-robin 0 
 \\_ 1:0:0:1 sdc  8:32   [ready]
 \\_ 2:0:0:1 sds  65:32  [ready]

create: mpath0 (36006016023e01200ed0aafd8727ddc11)
[size=34 GB][features="1 queue_if_no_path"][hwhandler="1 emc"]
\\_ round-robin 0 [prio=2]
 \\_ 2:0:1:2 sdab 65:176 [ready]
 \\_ 1:0:1:2 sdl  8:176  [ready]
\\_ round-robin 0 
 \\_ 1:0:0:2 sdd  8:48   [ready]
 \\_ 2:0:0:2 sdt  65:48  [ready]

create: mpath2 (36006016023e01200accd89df727ddc11)
[size=34 GB][features="1 queue_if_no_path"][hwhandler="1 emc"]
\\_ round-robin 0 [prio=2]
 \\_ 2:0:1:4 sdad 65:208 [ready]
 \\_ 1:0:1:4 sdn  8:208  [ready]
\\_ round-robin 0 
 \\_ 1:0:0:4 sdf  8:80   [ready]
 \\_ 2:0:0:4 sdv  65:80  [ready]

create: mpath3 (36006016023e01200627384e5727ddc11)
[size=34 GB][features="1 queue_if_no_path"][hwhandler="1 emc"]
\\_ round-robin 0 [prio=2]
 \\_ 2:0:1:5 sdae 65:224 [ready]
 \\_ 1:0:1:5 sdo  8:224  [ready]
\\_ round-robin 0 
 \\_ 1:0:0:5 sdg  8:96   [ready]
 \\_ 2:0:0:5 sdw  65:96  [ready]

create: mpath4 (36006016023e012004a38ee6f737ddc11)
[size=34 GB][features="1 queue_if_no_path"][hwhandler="1 emc"]
\\_ round-robin 0 [prio=2]
 \\_ 2:0:1:6 sdaf 65:240 [ready]
 \\_ 1:0:1:6 sdp  8:240  [ready]
\\_ round-robin 0 
 \\_ 1:0:0:6 sdh  8:112  [ready]
 \\_ 2:0:0:6 sdx  65:112 [ready]

create: mpath5 (36006016023e012004c38ee6f737ddc11)
[size=34 GB][features="1 queue_if_no_path"][hwhandler="1 emc"]
\\_ round-robin 0 [prio=2]
 \\_ 2:0:1:7 sdag 66:0   [ready]
 \\_ 1:0:1:7 sdq  65:0   [ready]
\\_ round-robin 0 
 \\_ 1:0:0:7 sdi  8:128  [ready]
 \\_ 2:0:0:7 sdy  65:128 [ready]



As you can see, two LUN:s are now missing. Also, the mpath0 and mpath1 devices are not mapped against the correct wwid. It looks like the multipath command has detected all LUN:s in a slightly different order and hence "overwritten" mpath0 and mpath1.

In order to get all LUN:s detected we need to remove the multipaths{} sections from the /etc/multipath.conf file and rerun the commands:

# multipath -F
# multipath -v2

Now all LUN:s are detected but the mpath0 and mpath1 devices are still incorrect. See output below:

create: mpath7 (36006016023e0120097c1f6b47807dc11)
[size=17 GB][features="1 queue_if_no_path"][hwhandler="1 emc"]
\\_ round-robin 0 [prio=2]
 \\_ 2:0:1:1 sdaa 65:160 [ready]
 \\_ 1:0:1:1 sdk  8:160  [ready]
\\_ round-robin 0 
 \\_ 1:0:0:1 sdc  8:32   [ready]
 \\_ 2:0:0:1 sds  65:32  [ready]

create: mpath0 (36006016023e01200ed0aafd8727ddc11)
[size=34 GB][features="1 queue_if_no_path"][hwhandler="1 emc"]
\\_ round-robin 0 [prio=2]
 \\_ 2:0:1:2 sdab 65:176 [ready]
 \\_ 1:0:1:2 sdl  8:176  [ready]
\\_ round-robin 0 
 \\_ 1:0:0:2 sdd  8:48   [ready]
 \\_ 2:0:0:2 sdt  65:48  [ready]

create: mpath1 (36006016023e01200aacd89df727ddc11)
[size=34 GB][features="1 queue_if_no_path"][hwhandler="1 emc"]
\\_ round-robin 0 [prio=2]
 \\_ 2:0:1:3 sdac 65:192 [ready]
 \\_ 1:0:1:3 sdm  8:192  [ready]
\\_ round-robin 0 
 \\_ 1:0:0:3 sde  8:64   [ready]
 \\_ 2:0:0:3 sdu  65:64  [ready]

create: mpath2 (36006016023e01200accd89df727ddc11)
[size=34 GB][features="1 queue_if_no_path"][hwhandler="1 emc"]
\\_ round-robin 0 [prio=2]
 \\_ 2:0:1:4 sdad 65:208 [ready]
 \\_ 1:0:1:4 sdn  8:208  [ready]
\\_ round-robin 0 
 \\_ 1:0:0:4 sdf  8:80   [ready]
 \\_ 2:0:0:4 sdv  65:80  [ready]

create: mpath3 (36006016023e01200627384e5727ddc11)
[size=34 GB][features="1 queue_if_no_path"][hwhandler="1 emc"]
\\_ round-robin 0 [prio=2]
 \\_ 2:0:1:5 sdae 65:224 [ready]
 \\_ 1:0:1:5 sdo  8:224  [ready]
\\_ round-robin 0 
 \\_ 1:0:0:5 sdg  8:96   [ready]
 \\_ 2:0:0:5 sdw  65:96  [ready]

create: mpath4 (36006016023e012004a38ee6f737ddc11)
[size=34 GB][features="1 queue_if_no_path"][hwhandler="1 emc"]
\\_ round-robin 0 [prio=2]
 \\_ 2:0:1:6 sdaf 65:240 [ready]
 \\_ 1:0:1:6 sdp  8:240  [ready]
\\_ round-robin 0 
 \\_ 1:0:0:6 sdh  8:112  [ready]
 \\_ 2:0:0:6 sdx  65:112 [ready]

create: mpath5 (36006016023e012004c38ee6f737ddc11)
[size=34 GB][features="1 queue_if_no_path"][hwhandler="1 emc"]
\\_ round-robin 0 [prio=2]
 \\_ 2:0:1:7 sdag 66:0   [ready]
 \\_ 1:0:1:7 sdq  65:0   [ready]
\\_ round-robin 0 
 \\_ 1:0:0:7 sdi  8:128  [ready]
 \\_ 2:0:0:7 sdy  65:128 [ready]

create: mpath8 (36006016023e01200ed7976c4717ddc11)
[size=17 GB][features="1 queue_if_no_path"][hwhandler="1 emc"]
\\_ round-robin 0 [prio=2]
 \\_ 1:0:0:0 sdb  8:16   [ready]
 \\_ 2:0:0:0 sdr  65:16  [ready]
\\_ round-robin 0 
 \\_ 1:0:1:0 sdj  8:144  [ready]
 \\_ 2:0:1:0 sdz  65:144 [ready]




Rebooting the server fixes the mpath device naming, but how should this be done without rebooting? Does one have to name _all_ LUN wwid:s in the /etc/multipath.conf file or use the bindings file? Considering some servers will have access to 20+ LUN:s they would like to avoid the manual process of setting up multipaths{} section with all these LUN:s.

How reproducible:

See above

Additional info:

I'm attacing the multipath.conf file we are using to this ticket.


This event sent from IssueTracker by sfernand  [Support Engineering Group]
 issue 164983

--- Additional comment from tao on 2008-09-04 08:22:39 EDT ---

File uploaded: multipath.conf
This event sent from IssueTracker by sfernand  [Support Engineering Group]
 issue 164983
it_file 120306

--- Additional comment from tao on 2008-09-04 08:22:42 EDT ---

Hello there,

Finally I have an answer to provide (of sorts anyway). Johan and I had
extensive conversations about this last week, and I set up a test system
to try this out on and between us, there is an answer.

You can do this, if you in /var/lib/multipath/bindnings specify the alias
you want and the WWID for the path. It seems that using the aliases is not
the correct way to do this, as this seems to happen:

Discover WWID 0001, create /dev/mpath/mpath0, discover WWID 0002, create
/dev/mpath/mpath1, discover WWID 0003 (but you have an alias saying you
want this to be mpath0), create /dev/mpath/mpath0 again.
As it is just symbolic links in /dev/mpath (the real device files are in
/dev/mapper) mpath0 that pointed at WWID 0001 now points to WWID 0003 and
no new link is created for the original path.

If you have a separate /var (separate from /), you may end up with a
problem that /var/lib/multipath/bindnings is not available at boot time
when it is needed by multipath and your path aliases still don't get set
up correctly. As of RHEL 4.6, there is an option in multipath.conf to have
the bindings file elsewhere, but before RHEL 4.6, you have to modify the
init scripts by hand to add the '-b /path/to/bindings' to the multipath
command.

I think this should answer the question.

Thanks!

/Anders


Internal Status set to 'Waiting on Customer'
Status set to: Waiting on Client
Resolution set to: 'Information Provided'

This event sent from IssueTracker by sfernand  [Support Engineering Group]
 issue 164983

--- Additional comment from tao on 2008-09-04 08:22:45 EDT ---

Taking this back to WoTech as there is a documentation point that we need
to consider in respect of "alias".

/Anders


Internal Status set to 'Waiting on Support'
Status set to: Waiting on Tech

This event sent from IssueTracker by sfernand  [Support Engineering Group]
 issue 164983

--- Additional comment from tao on 2008-09-04 08:22:47 EDT ---

Hello Friendly SEG Person,

Escalating this as a 'doc bug', but it's more of a "maybe we should
just point this out". It's a corner-case IMHO, but it does cause some
confusion. If you think a KBase is a better option, then we can go that
route.

For multipath, if you need to have specific paths turn up at a specific
name,
https://www.redhat.com/docs/manuals/csgfs/browse/4.6/DM_Multipath/multipath_consistent_names.html
say that if you use friendly names, you can put specific paths at a
specific name using the alias option. This is true, as long as you don't
use "mpathX" as your preferred alias. If you do, you have a very good
chance to end up in the situation where:

You discover WWID 0001, create /dev/mpath/mpath0, discover WWID 0002,
create /dev/mpath/mpath1, discover WWID 0003 (but you have an alias saying
you want this to be mpath0), create /dev/mpath/mpath0 again.
As it is just symbolic links in /dev/mpath (the real device files are in
/dev/mapper) mpath0 that pointed at WWID 0001 now points to WWID 0003 and
no new link is created for the original path.

Maybe we should in the documentation point out that using "mpathX" as
alias is not a good idea?

I'd like SEG's feedback and opinion on adjusting the docs or if there
are alternative means to resolve this (tweak multipath to check if a link
exists before writing a new one and move the original aside?). 

Thanks!

/Anders



Issue escalated to Support Engineering Group by: akarlsso.
Internal Status set to 'Waiting on SEG'

This event sent from IssueTracker by sfernand  [Support Engineering Group]
 issue 164983

--- Additional comment from tao on 2008-09-04 08:22:50 EDT ---

From: Johan Broman <jbroman>
Date: Sun, 18 May 2008 11:19:30 +0200
To: Anders Karlsson <akarlsso>
Subject: Re: (UPDATED) Issue #164983 (multipath -v2 command overwrites
aliases specified in multipath.conf)[SEG -
       	Storage/Support Engineering Group/IKEA]

Hi Anders!

Great. I think your summary is well written. However, the strange thing
is
that this works when using a bindings_file to name the devices mpathX and
so on, but not when using the multipath.conf file for assigning these
aliases. So no aliases are not "overwritten" when using the
bindings_file....
As well as updating the documentation I think there should be a more
consistent behaviour.

Cheers!
/Johan



This event sent from IssueTracker by sfernand  [Support Engineering Group]
 issue 164983

--- Additional comment from tao on 2008-09-04 08:22:52 EDT ---

From: Anders Karlsson <tak>
Date: Sun, 18 May 2008 14:11:28 +0200
To: Johan Broman <jbroman>
Subject: Re: (UPDATED) Issue #164983 (multipath -v2 command overwrites
       	aliases specified in multipath.conf)[SEG - Storage/Support
       	Engineering Group/IKEA]

* Johan Broman <jbroman> [20080518 11:19]:
> Hi Anders!
>
> Great. I think your summary is well written. However, the strange thing
is
> that this works when using a bindings_file to name the devices mpathX
and
> so on, but not when using the multipath.conf file for assigning these
> aliases. So no aliases are not "overwritten" when using the
> bindings_file....
> As well as updating the documentation I think there should be a more
> consistent behaviour.

Hi Johan,

I think I can explain that too.

When you use the bindings file, dm-mpio knows upfront what paths of
the ones it can see should have certain names. Reading in the bindings
file, it can see that it needs to name WWID xyz and WWID abc as mpath2
and mpath7 respectively, and leaves the appropriate gaps.

When you just specify aliases in multipath.conf, you are saying that
"when I find this path, irrespective of the the order of discovery, I
want this WWID to get this name". If you discover that path first,
great. If you discover it later, not so good. The alias is set on
discovery. This will work *as long as you do not use mpathX as the
alias*.

Think of it as "using the bindings_file is proactive, and setting
aliases in multipath.conf is reactive". You can achieve what you want
without using the bindings_file, you just have to specify a specific
alias for every path you will discover, not just the ones of specific
interest. :)

To get the functionality you are after, you'd need to merge the
bindings_file with multipath.conf, and I'm not sure that is a good
idea. What we should do, is document it better. This usecase is a
corner-case IMHO.

I hope this clarifies it better. (At least my understanding of how it
works, I am sure Bryn can correct me. :)

Thanks!

--
Anders


This event sent from IssueTracker by sfernand  [Support Engineering Group]
 issue 164983

--- Additional comment from tao on 2008-09-04 08:22:55 EDT ---

Hi again!

Yes, that certainly clarifies it all! :) Thanks for the info, we'll see
if the documentation needs updating.

Thanks!
/Johan


This event sent from IssueTracker by sfernand  [Support Engineering Group]
 issue 164983

--- Additional comment from tao on 2008-09-04 08:22:58 EDT ---

Hi Anders,

Sorry, this kept getting ignored for higher prio. issues, for so long.
Anyways, IMHO, this should be just a kbase article. If this really is a
document issue, I do not know where specifically this can fit in. We
document the use of aliases only in "Using Device-Mapper Multipath"
document[1] (which is possibly the best place for this).

Please let me know if you want me to escalate to documentation (besides
doing a kbase, which I think is a good idea, anyways).

regards,
- steve


[1]
https://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5.2/html/DM_Multipath/config_file_multipath.html


Internal Status set to 'Waiting on Support'

This event sent from IssueTracker by sfernand  [Support Engineering Group]
 issue 164983

--- Additional comment from tao on 2008-09-04 08:23:01 EDT ---

Yikes ! I just noticed this is for RHEL4. I don't know where in the RHEL4
docs
we specify the use of aliases, however, I'll escalate to docs. if
required.

let me know.
- steve



This event sent from IssueTracker by sfernand  [Support Engineering Group]
 issue 164983

--- Additional comment from tao on 2008-09-04 08:23:04 EDT ---

Hi Steven,

I think we should mention this in the docs, the very least as a caveat or
a "don't do that then" item. I'd expect most customers using aliases to be
more inventive than using "mpathX" aliases, but those that *do* do this
will end up raising issues on it.

I'll do the kbase bit but I'd like this to go in to the docs, so yes
please, punt it up to the docs team.

Cheers!

/Anders


Internal Status set to 'Waiting on SEG'

This event sent from IssueTracker by sfernand  [Support Engineering Group]
 issue 164983

--- Additional comment from tao on 2008-09-04 08:23:06 EDT ---

Hi Anders,

I am not too sure where this would fit in best in the RHEL4 docs, so
escalating to the device-mapper-multipath package itself. This probably
can be documented as a comment in the '/etc/multipath.conf' file of the
package or the in the man page for multipath (I don't see a conf file
specific man page in the package).

regards,
- steve


This event sent from IssueTracker by sfernand  [Support Engineering Group]
 issue 164983

--- Additional comment from pm-rhel on 2008-09-05 13:19:56 EDT ---

This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
release.