Bug 352421

Summary: Online resizing of a multipath map is impossible
Product: Red Hat Enterprise Linux 4 Reporter: Mimmus <viggiani>
Component: device-mapper-multipathAssignee: Ben Marzinski <bmarzins>
Status: CLOSED WORKSFORME QA Contact: Corey Marthaler <cmarthal>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.5CC: agk, akarlsso, andriusb, bmarzins, bstevens, cbolz, christophe.varoqui, daly, dwysocha, egoggin, jim.lester, junichi.nomura, kueda, lmb, mbroz, prockai, rlerch, rsarraf, tao, tom, tranlan, xdl-redhat-bugzilla
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
In RHEL 4.8 it is possible to do online resizing of multipath devices. The steps to do this are: 1. Resize your storage 2. Rescan your devices 3. run multipath multipath will notice the new path size, and correctly resize the multipath device. However, on some storage devices, it is necessary to unmap a LUN in order to resize it. In that case, any IO that is issued while the LUN is unmapped will fail, unless your multipath device is set to queue IO when no paths exist. To avoid this, you can run # dmsetup suspend --noflush <devicename> before doing step 1, and then # dmsetup resume <devicename> after doing step 1. This will ensure that no IO goes to the device while the LUN is unmapped.
Story Points: ---
Clone Of: Environment:
Last Closed: 2009-01-26 15:54:28 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 450897    

Description Mimmus 2007-10-25 14:59:10 UTC
Description of problem:
Online resizing of a multipath map is impossible.
Error is:
 device-mapper: reload ioctl failed: Invalid argument

Version-Release number of selected component (if applicable):
Red Hat 4.5

How reproducible:
Always

Steps to Reproduce:
1. Resize LUN by EMC Navisphere
2. Rescan four paths using:
 echo "1" > /sys/bus/scsi/drivers/sd/<h>\:<b>\:<t>\:<l>/block/device/rescan
3. Noticed in /var/log/messages that kernel "sees" new size only for two paths,
surely the active ones (Clariion is an active/passive array).
4. Trespassed LUN on the other storage processor and rescanned all paths a
second time: all sizes was updated
5. Run :
 # multipath -v 2
 device-mapper: reload ioctl failed: Invalid argument
6. I found no other way to make this work but umounting filesystem, deactivating
LVM and ran 'multipath -f ... && multipath -v2'

  
Actual results:
device-mapper: reload ioctl failed: Invalid argument

Expected results:
'multipath -v2' gives updated size

Additional info:
http://www.redhat.com/archives/dm-devel/2007-August/msg00188.html

Comment 2 Ryan Daly 2008-03-24 15:26:27 UTC
I'm experiencing this issue, as well.

I'm using RHEL 4.6.

: server17 42#; multipath -v 2 -d mpath0
reload: mpath0 (360a980004335434e654a48426b467552)
[size=320 GB][features="1 queue_if_no_path"][hwhandler="0"]
\_ round-robin 0 [prio=4]
 \_ 11:0:0:0    sdd 8:48  [active][ready]
 \_ 12:0:0:0    sde 8:64  [active][ready]

: server17 43#; multipath -v 2 mpath0
device-mapper: reload ioctl failed: Invalid argument

Comment 6 Tom Lanyon 2008-08-12 04:23:47 UTC
What's the status of this? Seemingly the same issue here in 4.6, kernel 2.6.9-55.0.9.ELsmp and dm-multipath 0.4.5-21.0.1.


Resized a FC LUN from 400 to 600 GB and the kernel picks up the size of the new devices:

# dmesg -c
SCSI device sde: 1258291200 512-byte hdwr sectors (644245 MB)
SCSI device sde: drive cache: write through


I tried removing and re-adding one path (deleted the dev node and re-added) but that didn't help as it can't add the 600 GB path to the 400 GB map:

# /sbin/multipath -ll oradbf
oradbf (3600508b400070f7d0000b00000310000)
[size=400 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [prio=110][active]
 \_ #:#:#:#  -   8:16  [failed][faulty]
 \_ 0:0:1:2 sde 8:64  [active][ready]
 \_ 1:0:0:2 sdh 8:112 [active][ready]
 \_ 1:0:1:2 sdk 8:160 [active][ready]


Multipath seems to *want* to increase the multipath map size, but can't:

# /sbin/multipath -v3 -d
0 1258291200 multipath 0 0 1 1 round-robin 0 4 1 8:64 5000 8:80 1000 8:112 5000 8:160 1000
set ACT_RELOAD: size change
device-mapper: reload ioctl failed: Invalid argument

# dmesg -c
device-mapper: device 8:64 too small for target
device-mapper: dm-multipath: error getting device
device-mapper: error adding target to table


Is there a workaround for this or perhaps its fixed in RHEL 5.x?

Comment 7 Mimmus 2008-08-12 09:39:23 UTC
It seems that a path was submitted mainstream but it was not reviewed by mantainer:
http://thread.gmane.org/gmane.linux.scsi/41623

Comment 8 Tom Lanyon 2008-08-12 10:27:16 UTC
(In reply to comment #7)
> It seems that a path was submitted mainstream but it was not reviewed by
> mantainer:
> http://thread.gmane.org/gmane.linux.scsi/41623

Hmm this patch was also the solution, it seems, for bug 455692 (https://bugzilla.redhat.com/show_bug.cgi?id=455692)

However, that bug doesn't seem to have a final resolution either.

Comment 9 Mimmus 2008-08-12 10:42:54 UTC
No, it is not solved (and it's a BIG issue in enterprise environment).
Here:
 https://www.redhat.com/archives/dm-devel/2008-August/msg00033.html
you can find a "unpolished, YMMV, no warranty recipe" to do online resize.
I didn't test it yet.

Suggestion: file a Request-for-Enhancement to Red Hat support, if you have a contract, Bugzilla is not the best place.

Comment 10 RHEL Program Management 2008-09-05 17:20:28 UTC
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
release.

Comment 12 Tom Lanyon 2008-11-11 04:01:38 UTC
Has there been any update on this yet?

This is a ridiculous bug in an enterprise system and we're having to use other operating systems until this is fixed.

Comment 13 Ben Marzinski 2009-01-23 00:41:42 UTC
*** Bug 479684 has been marked as a duplicate of this bug. ***

Comment 14 Ben Marzinski 2009-01-23 00:46:52 UTC
With the kernel fixed to allow online resize, this should just work now.
The steps should be:

1. rescan all of your devices (you may need to do the trespass to make sure that
all of the devices are resized)
2. run
# multipath

Now that the kernel can really change the size of the device while it's online, the multipath ioctl should work. This kernel change went into the 2.6.9-78.29.EL
kernel.

However, when testing this, I ran into some issues. They are described in bug 454872.  Once they are resolved. I'm going to close this bug as WORKSFORME

If you can reproduce the problem with the -78.29 or newer kernel, please reopen the bug.

Comment 15 Ben Marzinski 2009-01-23 21:30:21 UTC
The issue in 454872 has been resolved.  However there may need to be one more step necessary to resize your device without errors.  To resize some LUNs, you need
to temorarily unmap the LUN on your storage device.  In this case, while the LUN is unmapped, any IO that goes to the device will fail if the device isn't set to queue IO when all paths are down (by setting features to "1 queue_if_no_path" or setting no_path_retry to something other than "fail" in multipath.conf)

To avoid this problem on devices that don't queue when all paths are down. You can run dmsetup suspend --noflush <devicename> before you unmap the LUN, and then run
dmsetup resume <devicename> after you remap the LUN, and before you rescan the scsi device.

Comment 16 Ben Marzinski 2009-01-23 21:30:21 UTC
Release note added. If any revisions are required, please set the 
"requires_release_notes" flag to "?" and edit the "Release Notes" field accordingly.
All revisions will be proofread by the Engineering Content Services team.

New Contents:
In RHEL 4.8 it is possible to do online resizing of multipath devices. The steps to do this are:

1. Resize your storage
2. Rescan your devices
3. run multipath

multipath will notice the new path size, and correctly resize the multipath device.

However, on some storage devices, it is necessary to unmap a LUN in order to resize it.  In that case, any IO that is issued while the LUN is unmapped will fail, unless your multipath device is set to queue IO when no paths exist.

To avoid this, you can run
# dmsetup suspend --noflush <devicename>
before doing step 1, and then
# dmsetup resume <devicename>
after doing step 1.

This will ensure that no IO goes to the device
while the LUN is unmapped.

Comment 17 Ben Marzinski 2009-01-26 15:54:28 UTC
Instead of a release note. This information has simply been added to the errata description. I'm closing this bug, since no actual changes were necessary.