This service will be undergoing maintenance at 00:00 UTC, 2016-08-01. It is expected to last about 1 hours
Bug 484419 - FEAT: RHEL5.6: CCISS device-mapper-multipath support: missing sysfs attributes
FEAT: RHEL5.6: CCISS device-mapper-multipath support: missing sysfs attributes
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: device-mapper-multipath (Show other bugs)
5.6
All Linux
medium Severity medium
: rc
: 5.6
Assigned To: Ben Marzinski
Storage QE
OtherQA
: FutureFeature, OtherQA
: 249585 475177 (view as bug list)
Depends On: 514245
Blocks: 528587 557597 580580 484415 501986 522108 568828 600363
  Show dependency treegraph
 
Reported: 2009-02-06 12:51 EST by Bryn M. Reeves
Modified: 2011-05-04 04:36 EDT (History)
30 users (show)

See Also:
Fixed In Version: device-mapper-multipath-0.4.7-38.el5
Doc Type: Enhancement
Doc Text:
CCISS devices are now supported.
Story Points: ---
Clone Of: 484415
Environment:
Last Closed: 2011-01-13 18:01:23 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
Output of multipath -v3 (1.27 KB, text/plain)
2010-06-21 11:41 EDT, Oliver Hookins
no flags Details
multipath.conf for this scenario (589 bytes, application/octet-stream)
2010-11-18 11:00 EST, Martin George
no flags Details

  None (edit)
Description Bryn M. Reeves 2009-02-06 12:51:57 EST
+++ This bug was initially created as a clone of Bug #484415 +++

Description of problem:
The multipath-tools components rely on being able to retrieve vendor/product information from sysfs in order to match storage controllers to their configurations.

The CCISS driver does not export these attributes.

Version-Release number of selected component (if applicable):
All RHEL5 kernels.

How reproducible:
100%

Steps to Reproduce:
1. Install device-mapper-multipath on a system with dual CCISS controllers
2. Attempt to configure multipath maps (e.g multipath -v3)

  
Actual results:
# multipath -v6
load path identifiers cache
#
# all paths in cache :
#
path cciss!c0d0 not found in pathvec

===== path info cciss!c0d0 (mask 0x1f) =====
path cciss!c1d0 not found in pathvec

===== path info cciss!c1d0 (mask 0x1f) =====
path cciss!c1d1 not found in pathvec

===== path info cciss!c1d1 (mask 0x1f) =====
path cciss!c2d0 not found in pathvec

===== path info cciss!c2d0 (mask 0x1f) =====
path cciss!c2d1 not found in pathvec

===== path info cciss!c2d1 (mask 0x1f) =====

Expected results:
Multipath maps detected and configured.

Additional info:
CCISS upstream does not support these attributes either. Patches adding them were posted twice to LKML back in 2007. On the first post a number of problems were pointed out but the repost addressing these does not appear to have gone anywhere:

http://lkml.indiana.edu/hypermail/linux/kernel/0711.2/1026.html
http://lkml.indiana.edu/hypermail/linux/kernel/0712.1/2724.html
Comment 3 Tom Coughlan 2009-02-11 08:16:12 EST
Mike, 

Do you suppose you could try again to get this upstream? The we can backport it to RHEL 5. 

Tom
Comment 4 Bryn M. Reeves 2009-02-11 09:16:02 EST
Maybe it's possible/worthwhile to separate out the two attributes (vendor/product) device-mapper-multipath needs from the rest of the sysfs attributes patch?
Comment 5 Mike Miller (OS Dev) 2009-02-11 10:11:58 EST
Is that all that's needed, the vendor and product attributes? Separating these 2 out is probably the easiest way to get it accepted.
Comment 6 Bryn M. Reeves 2009-02-11 10:34:17 EST
Afaik yes - those are the only attributes multipath needs to retrieve from sysfs that are missing for CCISS (it's just to match the config entries in multipath.conf/hwtable.c).
Comment 7 Andrew Patterson 2009-03-03 12:23:13 EST
Where do you expect vendor and model to show up in the sysfs tree.  I currently have a patch that has these entries in:

/sys/class/cciss_logical_drive/c#d#
Comment 8 Bryn M. Reeves 2009-03-03 12:32:06 EST
They should appear in files named "vendor" and "product" inside the device's device directory.

E.g., this is how the accessor functions in RHEL-5's multipath-tools look today:

declare_sysfs_get_str(vendor, "%s/block/%s/device/vendor");
declare_sysfs_get_str(model, "%s/block/%s/device/model");
Comment 9 Tomas Henzl 2009-03-18 12:03:06 EDT
*** Bug 249585 has been marked as a duplicate of this bug. ***
Comment 10 Tomas Henzl 2009-04-03 11:05:06 EDT
Mike,
this will probably make it into RHEL4, so we now should have it also for RHEL5.
Please post an appropriate patch for this here.
Thanks,
Tomas
Comment 11 Mike Miller (OS Dev) 2009-04-21 16:56:34 EDT
Does everything work as expected? I'm getting reports around that the multi-path is not working, i.e., no devices showing up.
Comment 12 Tomas Henzl 2009-04-22 04:44:21 EDT
(In reply to comment #11)
> Does everything work as expected? I'm getting reports around that the
> multi-path is not working, i.e., no devices showing up.
I'll test it (ask someone what is missing) in near future.
Comment 13 RHEL Product and Program Management 2009-04-22 04:49:12 EDT
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
release.
Comment 14 Tomas Henzl 2009-04-29 05:08:27 EDT
(In reply to comment #11)
> Does everything work as expected? I'm getting reports around that the
> multi-path is not working, i.e., no devices showing up.  

It looks like we need not only vendor/product, but also the bus attribute.
Thanks, Tomas
Comment 15 Andrew Patterson 2009-04-29 11:53:52 EDT
This patch has the bus attributes too (not backported to rhel 4 yet):

http://marc.info/?l=linux-kernel&m=123912749517010&w=2

Mike Miller has a slightly newer version that addresses Andrew Morton's review comments.
Comment 16 Tomas Henzl 2009-05-05 06:42:12 EDT
> Mike Miller has a slightly newer version that addresses Andrew Morton's review
> comments.  

Mike, please post your version so we can start testing the newest patch.
Thanks,
Tomas
Comment 17 Tomas Henzl 2009-05-14 13:19:32 EDT
(In reply to comment #15)
> This patch has the bus attributes too (not backported to rhel 4 yet):
> 
> http://marc.info/?l=linux-kernel&m=123912749517010&w=2

Andrew, are you sure the bus attribute is there? I've seen only
model, number, revision and not bus. See the patch in bz484415 https://bugzilla.redhat.com/attachment.cgi?id=342323
Comment 18 Andrew Patterson 2009-05-14 13:34:20 EDT
The patchset in http://marc.info/?l=linux-kernel&m=123912749517010&w=2 is not in
rhel 4.8.  My hope was that it would make it there, but it seems an earlier patch by Chase Maupin was taken instead with some later modifications.
Comment 19 Tomas Henzl 2009-05-14 13:45:21 EDT
(In reply to comment #18)
> The patchset in http://marc.info/?l=linux-kernel&m=123912749517010&w=2 is not
> in
> rhel 4.8.  My hope was that it would make it there, but it seems an earlier
> patch by Chase Maupin was taken instead with some later modifications.  

Andrew,please could you provide a patch with vendor/product/bus attribute for multipath ?
Our latest sources are here http://people.redhat.com/dzickus/el5/ - the biggest number.
Thanks.
Comment 20 Andrew Patterson 2009-05-14 16:12:17 EDT
Mike Miller is going to backport the http://marc.info/?l=linux-kernel&m=123912749517010&w=2 patches.
Comment 21 Tomas Henzl 2009-05-26 07:07:13 EDT
We are too late, so I'm moving this to RHEL5.5
Comment 22 Mike Miller (OS Dev) 2009-05-27 15:25:23 EDT
I'm hitting a dependency I can't resolve. 

[root@duke] rpm -i kernel-2.6.18-150.el5.src.rpm
error: Failed build dependencies:
        hmaccalc is needed by kernel-2.6.18-150.el5.x86_64

Where can I find the package containing hmaccalc?
Comment 23 Tom Coughlan 2009-05-27 16:07:27 EDT
(In reply to comment #22)

> Where can I find the package containing hmaccalc?  

I just mailed it to you.
Comment 24 Andrew Patterson 2009-06-01 16:23:39 EDT
I am going to do this backport for Mike.  I need an ia64 version of the hmaccalc
rpm.

Andrew
Comment 25 Andrew Patterson 2009-06-01 16:31:58 EDT
BTW, Andrew Morton accepted this patch in his kernel tree.
Comment 26 Tomas Henzl 2009-06-02 06:28:45 EDT
(In reply to comment #24)
> I am going to do this backport for Mike.  I need an ia64 version of the
> hmaccalc
> rpm.
> 
> Andrew  

I think that you get the error message when you start rpmbuild and not rpm right?
If it is so then can add argument "--without fips" to the command line.
example for x86 : rpmbuild -bp --target=x86_64 --without fips kernel-2.6.spec

I'll also look for hmaccalc and send it to you.
Tomas
Comment 28 Ronald Pacheco 2009-07-07 08:09:13 EDT
Moving to RHEL 5.5 since this was not resolved prior to the beta.
Comment 29 Tomas Henzl 2009-10-22 09:51:40 EDT
Andrew,
It looks like upstream accepted the vendor/product, but we also need the 'bus' attribute for multipath.

Thanks, Tomas 

(In reply to comment #25)
> BTW, Andrew Morton accepted this patch in his kernel tree.
Comment 30 Tomas Henzl 2009-11-30 08:37:29 EST
Bryan,
most of the attributes - model,vendor,id,rev already is in RHEl5. Please let me know if this is enough or if we really need some additional fields.
Thanks, Tomas
Comment 31 Tomas Henzl 2010-01-15 11:21:22 EST
Moving it to RHEL5.6 it will be solved there when we know what else is needed.
Comment 32 Sandy Garza 2010-01-27 18:03:57 EST
(In reply to comment #31)
> Moving it to RHEL5.6 it will be solved there when we know what else is needed.    

HP agrees to defer this to RHEL 5.6.
Comment 33 Bryn M. Reeves 2010-02-22 06:28:21 EST
I'm not able to test myself (no hardware) and this BZ appears to have lost its IT tickets - we'll likely need to rely on HP to test this change in 5.6.
Comment 34 Ben Marzinski 2010-05-12 13:58:02 EDT
*** Bug 475177 has been marked as a duplicate of this bug. ***
Comment 35 Tomas Henzl 2010-05-13 09:02:20 EDT
Sandy,
this might be resolved in between, it probably needs only some testing with hw I don't have. Please verify that and if a patch is still needed for this post it.
Comment 36 Oliver Hookins 2010-06-14 05:36:38 EDT
I have compatible hardware that could be used to test this. Please let me know what I need to do.
Comment 37 Tomas Henzl 2010-06-14 11:00:17 EDT
(In reply to comment #36)
> I have compatible hardware that could be used to test this. Please let me know
> what I need to do.    

Install RHEL5.5 and let us know if multipath works for you. I think the id's are all in so it should work just fine. 
Here should be some description http://www.redhat.com/docs/manuals/enterprise/RHEL-5-manual/en-US/RHEL510/pdf/DM_Multipath.pdf
Comment 38 Oliver Hookins 2010-06-21 05:15:33 EDT
I've just installed RHEL5.5 and it seems to have the same problems I have already seen:
 * scsi_id has undocumented requirement of the -s parameter
 * scsi_id -s parameter doesn't seem to work properly anyway
 * even when overriding the getuid_callout with a script that returns the correct device UIDs to use with multipath, output of `multipath -ll -v3` still ignores the actual devices with output like the following

.
.
.
cciss!c0d0: exception-listed
cciss!c0d0: not found in pathvec
cciss!c0d0: mask = 0x5
.
.
.

For all of the paths.
Comment 39 Tomas Henzl 2010-06-21 06:42:32 EDT
Oliver,
thanks for the report.
Comment 40 Tomas Henzl 2010-06-21 06:45:00 EDT
Ben,
could you please look into this, let us know what else should be added to the cciss driver to make it work with multipath.
Comment 41 Bryn M. Reeves 2010-06-21 09:00:52 EDT
RHEL5.5 has the cciss_id tool which should be used instead for CCISS controllers:

# which cciss_id
/sbin/cciss_id
# rpm -qf $(which cciss_id)
device-mapper-multipath-0.4.7-34.el5_5.1

cciss!c0d0: exception-listed

Shows that this is matching a provided blacklist exceptions regex.

Can you post the multipath.conf that you are using and the full output from "multipath -v4"?
Comment 42 Oliver Hookins 2010-06-21 09:31:50 EDT
Certainly, /sbin/cciss_id fixes the issue with getting a UID which the current version of scsi_id cannot do:

[root@localhost ~]# cciss_id /dev/cciss/c0d0
3600c0ff000da5e68632a124c01000000

However even when using this in the multipath.conf getuid_callout there is still a problem actually configuring the paths, which seems to point to a problem in multipath itself:

[root@localhost ~]# multipath -ll -v3
cciss!c0d0: exception-listed
cciss!c0d0: not found in pathvec
cciss!c0d0: mask = 0x5
.
.
.
Comment 43 Ben Marzinski 2010-06-21 11:33:19 EDT
Could you please run 

# multipath -v3

and attach the output to the bugzilla
Comment 44 Oliver Hookins 2010-06-21 11:41:13 EDT
Created attachment 425676 [details]
Output of multipath -v3
Comment 46 Tom Coughlan 2010-07-06 10:11:45 EDT
It appears as though the cciss driver and the cciss_id callout may provide the needed information. I'll re-assign this to Ben to look at it from the multipath side. 

Ben probably does not have access to this hardware, so please relay any information he requests as needed. 

Tom
Comment 53 Ben Marzinski 2010-09-01 16:57:53 EDT
Nevermind. I see that it's at:

/sys/block/<devname>/device/cciss<X>/c<X>d<Y>/model
Comment 54 Ben Marzinski 2010-09-02 23:56:49 EDT
device-mapper-multipath can now create multipath devices on top of cciss devices.
Comment 56 Ben Marzinski 2010-09-09 13:34:24 EDT
I've just commited the fix. I haven't built a package with it yet.  If you'd like one to try out, I can build one today.
Comment 57 Oliver Hookins 2010-09-09 17:41:39 EDT
I would definitely like to try out the fix.
Comment 59 Ben Marzinski 2010-09-28 09:37:25 EDT
Sorry. I put them up but forgot to post a link in the bugzilla.

you can get them at:

http://people.redhat.com/~bmarzins/device-mapper-multipath/rpms/x86_64/

or

http://people.redhat.com/~bmarzins/device-mapper-multipath/rpms/i386/

The packages you want are:

device-mapper-multipath-0.4.7-38.el5.i386.rpm
device-mapper-multipath-debuginfo-0.4.7-38.el5.i386.rpm
kpartx-0.4.7-38.el5.i386.rpm
Comment 60 Barry Donahue 2010-11-12 12:33:09 EST
Verified on RHEL5.6-Server-20101110.0, device-mapper-multipath-0.4.7-39.el5.
Comment 61 Martin George 2010-11-18 10:00:44 EST
I'm hitting dm-multipath + cciss issues while running RHEL 5.6 on an HP Blade. I was trying to configure dm-multipath on the NetApp LUNs (and not on the cciss devices) mapped to my RHEL 5.6 host, but always end up with the error mentioned below:

Config:
RHEL 5.6 Snap1 (2.6.18-231.el5)
cciss driver v3.6.22-RH1

Running any multipath command always throws up the following error:
'can't get driver link: No such file or directory'. 

Running an strace on the multipath command shows the issue at:

...
stat("/sys/block/cciss!c0d0/device/cciss0/c0d0/vendor", 0x7fffc71c4b00) = -1 ENOENT (No such file or directory)
ioctl(3, DM_LIST_DEVICES, 0x110e6460)   = 0
ioctl(3, DM_TABLE_STATUS, 0x110ea470)   = 0
ioctl(3, DM_TABLE_STATUS, 0x110ea470)   = 0
close(3)                                = 0
write(1, "can't get driver link: No such f"..., 49can't get driver link: No such file or directory
) = 49

Seems dm-multipath throws this error when it fails to get the cciss vendor sysfs attribute. I tried blacklisting the local cciss devices in the multipath.conf, but to no avail i.e. the error is seen irrespective of blacklisting the cciss devices.

Is this a known issue? Should I file a new bug for this?
Comment 62 Ben Marzinski 2010-11-18 10:19:34 EST
could you attach your /etc/multipath.conf?  Blacklisting cciss devices should solve the issue.  The multipath command works correctly otherwise, right?
Comment 63 Martin George 2010-11-18 11:00:15 EST
Created attachment 461328 [details]
multipath.conf for this scenario
Comment 64 Ben Marzinski 2010-11-18 11:37:30 EST
To be honest, I have no idea what

[p[0-9]*]

matches in a POSIX extended regular expression.
could you try replacing

devnode "^cciss!c[0-9]d[0-9]*[p[0-9]*]"

with something simpler, like

devnode "^cciss.*"

or

devnode "^cciss!c[0-9]d.*"

and see if that works.
Comment 65 Linqing Lu 2010-11-19 05:22:30 EST
Hi Barry,
Since we have a similar bug in hand, could give us some detail information about your verification with this bug, like testing steps?
Thanks very much~
Comment 66 Barry Donahue 2010-11-19 09:09:01 EST
I installed the release on a system with CCISS disks and verified that the expected attributes were in sysfs and where they were expected. Then run multipath and verify no errors are generated.

   I supposed this bug(In reply to comment #65)
> Hi Barry,
> Since we have a similar bug in hand, could give us some detail information
> about your verification with this bug, like testing steps?
> Thanks very much~

I installed the release on a system with CCISS disks and verified that the expected attributes were in sysfs and where they were expected. Then run multipath and verify no errors are generated.

   I supposed this bug should have a precise test case from devel indicating exactly what sysfs attributes were added.
Comment 67 Martin George 2010-11-19 13:38:32 EST
(In reply to comment #64)
> could you try replacing
> 
> devnode "^cciss!c[0-9]d[0-9]*[p[0-9]*]"
> 
> with something simpler, like
> 
> devnode "^cciss.*"
> 
> or
> 
> devnode "^cciss!c[0-9]d.*"
> 
> and see if that works.

Yes, that works. Blacklisting cciss entries using "^cciss.*" & "^cciss!c[0-9]d.*" both work fine. 

Actually the original entry worked fine in previous RHEL5 kernels. So it does look like the multipath.conf parsing is more strict now in RHEL 5.6.
Comment 69 Ben Marzinski 2010-11-24 17:10:40 EST
I have some test packages that will hopefully fix this. You can get them at

http://people.redhat.com/bmarzins/device-mapper-multipath/rpms/x86_64/

or

http://people.redhat.com/bmarzins/device-mapper-multipath/rpms/i386/
Comment 70 Eva Kopalova 2010-11-30 06:47:17 EST
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
CCISS devices are now supported.
Comment 73 Chris Ward 2010-12-02 10:26:13 EST
Reminder! There should be a fix present for this BZ in snapshot 3 -- unless otherwise noted in a previous comment.

Please test and update this BZ with test results as soon as possible.
Comment 74 Alan Pevec 2011-01-05 04:44:53 EST
Is this verified or not?
Comment 79 errata-xmlrpc 2011-01-13 18:01:23 EST
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHEA-2011-0074.html

Note You need to log in before you can comment on or make changes to this bug.