Bug 537514 - [LSI 5.5 feat] make scsi_dh_activate asynchronous to address the slower lun failovers with large number of luns
Summary: [LSI 5.5 feat] make scsi_dh_activate asynchronous to address the slower lun f...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: kernel
Version: 5.5
Hardware: All
OS: Linux
high
high
Target Milestone: rc
: 5.5
Assignee: Rob Evers
QA Contact: Red Hat Kernel QE team
URL:
Whiteboard:
: 523668 (view as bug list)
Depends On: 537257
Blocks: 496328 533941 5.5TechNotes-Updates
TreeView+ depends on / blocked
 
Reported: 2009-11-13 23:24 UTC by Babu Moger
Modified: 2010-11-22 23:28 UTC (History)
15 users (show)

Fixed In Version:
Doc Type: Enhancement
Doc Text:
Fix enabling faster 'rdac' device handler device activation. Removes long delays on rdac device path activation noticed when paths to high number of luns activate simultaneously, such as a cable-pull in an active-passive multi-path environment.
Clone Of:
Environment:
Last Closed: 2010-03-30 07:25:01 UTC
Target Upstream Version:


Attachments (Terms of Use)
Make scsi_dh_activate async (7.15 KB, patch)
2009-12-16 20:41 UTC, Chandra Seetharaman
no flags Details | Diff
Batch up MODE SELECTS in rdac scsi hardware handler (6.84 KB, patch)
2009-12-16 20:42 UTC, Chandra Seetharaman
no flags Details | Diff
Make scsi_dh_activate async (8.08 KB, patch)
2009-12-18 19:24 UTC, Chandra Seetharaman
no flags Details | Diff
Batch up MODE SELECTs in rdac scsi hardware handler (6.84 KB, patch)
2009-12-18 19:33 UTC, Chandra Seetharaman
no flags Details | Diff


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2010:0178 normal SHIPPED_LIVE Important: Red Hat Enterprise Linux 5.5 kernel security and bug fix update 2010-03-29 12:18:21 UTC

Description Babu Moger 2009-11-13 23:24:15 UTC
Description of problem:

This bugzilla is to address the limitation in the current linux kernel with
scsi device handlers. 

Currently, scsi device handlers process path activation(activate_path)
synchronously. This leads to a lot of time delay when more than 100 luns are
involved. In case of rdac device handler’s path activation involves sending the
mode select command to transfer ownership of the specific lun. Each mode select
can take couple of seconds or more. This leads to a lot of time delay when more
than 100 luns are involved. 

This time delay can be avoided if we can do the activations asynchronously.
By making scsi_dh_activate() async, we can give the device handlers an
opportunity to decide on how to send the device activation down the wire
to make the turn-around time faster. They can send the commands asynchronously
or send them in batches. More details can be found at
http://marc.info/?l=linux-scsi&m=125614219830795&w=2

Following patches are submitted to upstream to address this problem.
http://marc.info/?l=linux-scsi&m=125614220030809&w=2
http://marc.info/?l=linux-scsi&m=125614219530777&w=2
http://marc.info/?l=linux-scsi&m=125614219630790&w=2
http://marc.info/?l=linux-scsi&m=125614220430822&w=2

These are expected to make it to 2.6.33 kernel. 
http://git.kernel.org/?p=linux/kernel/git/jejb/scsi-misc-2.6.git;a=commit;h=87f5f7800a3477b20c2a7760d5a150da572424d3

We would like to add these patches to RHEL 5.5 kernel. We will work to port/test
these patches.

Patches will follow soon…


Version-Release number of selected component (if applicable):


How reproducible:

Always if more than 100 luns are involved.

Steps to Reproduce:
1. Install OS
2. Present 100 or more luns to the host with two paths(one active, one passive)
3. Configure device mapper with rdac device handler
4. start i/o on all the 100 luns
5. Pull the cable to fail one path on all the luns.
  
Actual results:
Device mapper takes a while to trasfer the all the luns to surving path. It
could table about 5 to 10 minutes to trasfer all the luns.

Expected results:
We expect device mapper to complete the failovers in few seconds(or in a minute)

Additional info:

Comment 1 Andrius Benokraitis 2009-11-16 06:15:50 UTC
Babu - as stated in email, this feature will be evaluated if time and resources allow, since it is a late request for RHEL 5.5.

Comment 3 Andrius Benokraitis 2009-11-30 22:56:37 UTC
Babu - can you please create a similar RHEL 6 bugzilla with this info to get this added there as well?

Comment 4 Babu Moger 2009-12-01 04:28:41 UTC
Andrius,
Yes, We have already created a bugzilla for RHEL 6. The bugzilla number is 537257.

Comment 5 Andrius Benokraitis 2009-12-01 15:45:40 UTC
thanks Babu! Updated!

Comment 8 Tom Coughlan 2009-12-02 16:27:40 UTC
*** Bug 523668 has been marked as a duplicate of this bug. ***

Comment 9 Tom Coughlan 2009-12-02 16:35:51 UTC
The patches do not apply to RHEL 5.4. They depend on additional upstream
changes in the SCSI midlayer and rdac that are not present in RHEL 5. 

We will require these patches to be backported to a recent RHEL 5 version
before we can take them. Considering how late this arrived for 5.5, we it would
be most helpful for the requestor to provide these backports. 

Tom

Comment 10 Chandra Seetharaman 2009-12-14 22:15:41 UTC
Dear RedHat,

I ported the changes to RHEL 5.5 tree (2.6.18-176) and it compiles clean.

But, there is an issue.

SInc ethe data structure scsi_device_handler has changed, some of the scsi_dh internal functions looks different w.r.t kABI compliance.
The functions are:
  store_scsi_dh_data
  scsi_unregister_device_handler
  retrieve_scsi_dh_data
  scsi_register_device_handler
  scsi_dh_activate

Let me know if it is ok to have this breakage. 

If it is not ok, I will work on making it ABI compliant.

Thanks

Comment 11 Chandra Seetharaman 2009-12-15 18:02:56 UTC
RedHat,

Please advice on the ABI issue raised in the previous comment.

Comment 12 Rob Evers 2009-12-15 21:05:24 UTC
Jon,

Can you comment on the kabi issue raised in comment 10?

Thanks, Rob

Comment 13 Mike Christie 2009-12-16 00:10:40 UTC
Are those functions under kabi? My scsi_dh_emc patch modifies the scsi_device_handler struct, so it would break kabi if the symbols were listed. I do not think check_kabi is spitting out any errors though (it is hard to tell because there are already errors before the patch).

IBM were you seeing new check_kabi errors?

Comment 14 Chandra Seetharaman 2009-12-16 14:04:55 UTC
Mike,

I just compare Module.symvers (that is generated) with and without the patch and it does show errors.

It shows errors with your scsi_dh_emc patch too.

Where do I get check_kabi script/tool ?

Comment 15 Mike Christie 2009-12-16 16:18:36 UTC
Will send you the tools.

Comment 16 Chandra Seetharaman 2009-12-16 20:40:12 UTC
Ported the patches, tested them and verified that they work as expected.

Used the kabi tools Mike sent and verified that the two patches (below) does not break any kABI.

Comment 17 Chandra Seetharaman 2009-12-16 20:41:07 UTC
Created attachment 378839 [details]
Make scsi_dh_activate async

Comment 18 Chandra Seetharaman 2009-12-16 20:42:19 UTC
Created attachment 378840 [details]
Batch up MODE SELECTS in rdac scsi hardware handler

Comment 19 Chandra Seetharaman 2009-12-16 20:43:40 UTC
Both the above patches applies cleanly on top of 2.6.18-180

Comment 20 Rob Evers 2009-12-18 19:20:49 UTC
Chandra,

Can you update the patch with the problem found in bz537527 comment 14 if it applies and post the fix to linux-scsi?

https://bugzilla.redhat.com/show_bug.cgi?id=537257#c14

Thanks, Rob

Comment 21 Chandra Seetharaman 2009-12-18 19:24:07 UTC
Created attachment 379276 [details]
Make scsi_dh_activate async

This patch is same as the above one but applies cleanly on 2.6.18-182

Comment 22 Chandra Seetharaman 2009-12-18 19:33:32 UTC
Created attachment 379279 [details]
Batch up MODE SELECTs in rdac scsi hardware handler

Fixed a suggestion Mike Christie provided in Bug # 537257.

This also applies cleanly on 2.6.18-182

Comment 23 Chandra Seetharaman 2009-12-18 19:36:35 UTC
(In reply to comment #20)
> Chandra,
> 
> Can you update the patch with the problem found in bz537527 comment 14 if it
> applies and post the fix to linux-scsi?
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=537257#c14
> 
> Thanks, Rob  

Hi Rob,

New patch provided above handles the issue you pointed.

Thanks

Comment 24 Andrius Benokraitis 2010-01-07 19:36:20 UTC
Chandra - has this been posted upstream?

Comment 25 Babu Moger 2010-01-07 21:54:30 UTC
Yes, I have already posted the fix for this patch to upstream. Here is the link..
http://marc.info/?l=linux-scsi&m=126271209308764&w=2

All the other patches mentioned in the description are already in upstream.

Comment 26 Andrius Benokraitis 2010-01-27 16:16:06 UTC
Babu - has this patch been /committed/ upstream?

Also, bug 537257 has the rhel 6 patchset that seems to committed already. Do the rhel 5 and 6 patchsets match?

Comment 27 Babu Moger 2010-01-27 16:54:17 UTC
Andrius, 
Yes all these patches have been committed to upstream. It was included in 2.6.33-rc1.  Here are the commit id with links..

1) scsidh_activate interface changes
 http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=3ae31f6a7b6e442fc6a92f29330fbad230dc3992

2) rdac hardware handler changes
 http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=970f3f47e7c97c0bfe9f91356943b55ac389cb1d

3) hp hardware handler changes 
 http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=4e2ef86cd5ce057b60acea33bb71c06676e71888

4) alua hardware handler changes. 
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=96e6586556dfa80112f42895be93c561582d9930


For bug 537514 (RHEL 5.5)
- Chandra ported the patches 1 and 2 to RHEL 5.5. Because of time and resource constraints we did not port patch 3 and 4.  

For bug 537257 (RHEL 6)
- I(Babu) ported the all the 4 patches to RHEL 6.

Please let me know if you have any more questions..
Thanks

Comment 28 Babu Moger 2010-01-27 22:42:13 UTC
Andrius,
Now, I remember. There is one minor patch waiting for upstream commit. 
Here is the link.. http://marc.info/?l=linux-scsi&m=126463085821695&w=2

This is pretty new patch(three weeks old). Probably we might have to wait some more time to get this included in upstream. 

Thanks
Babu

Comment 29 Andrius Benokraitis 2010-01-27 23:42:28 UTC
If this patch is holding up everything, should we just not include it and move forward?

Comment 30 Chandra Seetharaman 2010-01-28 01:08:56 UTC
Hi Andrius,

It is a one line patch changing GFP_KERNEL to GFP_NOIO in a kzalloc() call.

As a matter of fact, this was a result of review made by Mike Christie for the patches we submitted for bug #537257 (See Comment 14 - https://bugzilla.redhat.com/show_bug.cgi?id=537257#c14)

IMHO, you can include it as I do not see there will be any issues of it being accepted upstream.

Comment 31 Rob Evers 2010-01-28 14:43:21 UTC
(In reply to comment #30)
> Hi Andrius,
> 
> It is a one line patch changing GFP_KERNEL to GFP_NOIO in a kzalloc() call.
> 
> As a matter of fact, this was a result of review made by Mike Christie for the
> patches we submitted for bug #537257 (See Comment 14 -
> https://bugzilla.redhat.com/show_bug.cgi?id=537257#c14)
> 
> IMHO, you can include it as I do not see there will be any issues of it being
> accepted upstream.    

I agree that this particular issue should not hold up this patch, in fact, we want the change, as Chandra suggests.

Comment 33 Narayanan D 2010-01-29 10:17:30 UTC
Business Justification for inclusion of this issue in RHEL 5.5:

This feature is required to support RHEL 5.5 with Hoggs storage enclosures.This storage enclosures is scheduled to release in June 2010. RHEL 5.5 support with this storage without this feature.

Comment 34 Narayanan D 2010-01-29 10:55:21 UTC
*RHEL 5.5 support on this storage without this feature is not possible

Comment 35 Narayanan D 2010-02-02 16:21:11 UTC
Patches 1 and 2 listed in the bugzilla are sufficient for Dell to test out this feature with our storage enclosure.

Comment 38 Rob Evers 2010-02-05 17:22:44 UTC
Technical note added. If any revisions are required, please edit the "Technical Notes" field
accordingly. All revisions will be proofread by the Engineering Content Services team.

New Contents:
Fix enabling faster 'rdac' device handler device activation.  Removes long delays on rdac device path activation noticed when paths to high number of luns activate simultaneously, such as a cable-pull in an active-passive multi-path environment.

Comment 43 Marizol Martinez 2010-02-12 18:49:09 UTC
Partners -- This BZ was approved for *snapshot 1*, so please wait for snapshot 1 code to test it.

Comment 46 Vijay Kumar 2010-03-15 12:34:20 UTC
I verified with snapshot1 and it works good. During failover, only two modeselect was issued to transfer ownership of 100 volumes. The ownership of 100 LUN's was transferred in just 25 seconds.

Thanks.

Comment 48 errata-xmlrpc 2010-03-30 07:25:01 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2010-0178.html


Note You need to log in before you can comment on or make changes to this bug.