Bug 455692

*** Bug 457129 has been marked as a duplicate of this bug. ***

This mail thread might be of interest:

http://www.linux-archive.org/device-mapper-development/138757-status-resizing-growing-dm-multipath-devices-fly.html

There is now a new multipathd command, "resize map <mapname>", which will do the multipath work to resize a block device.  There is one problem. Because of a kernel device-mapper issue, you cannot resize a device-mapper device with no_flush on.  This means that you will flushed any queued IOs when you suspend the device to change it's size.  This will only cause problems if user attempts to resize when all paths are down, and the device is set to queue_if_no_path.

*** Bug 445262 has been marked as a duplicate of this bug. ***

Ben,

I would like to test this feature. I looked upstream and at sourceware, but I don't see the resize stuff anywhere.

I'd like to test this feature too! Any test-packages for dm-multipath available?

There are test packages available at:
http://people.redhat.com/rpeterso/Experimental/RHEL5.x/dm-multipath/

Thanks! Although I'd need i386 binary.. could you please upload also i386 package, and/or srpm?

O.k. the i386 rpms are up as well.

Thanks.

I just tried it but it doesn't seem to work for me.. 

First I resized (i)SCSI devices, and that went OK. "/proc/partitions" and kernel/dmesg show the new/bigger size for SCSI devices.

Then I tried resizing dm-multipath device:

# multipathd -k"resize map mpath-resize-test"
ok

But 'mpath-resize-test' device is still same/old size after that.. based on "multipath -ll" output.

Any idea how to get multipath-device to see the new size? 
Is there something else I need to do/run? 

I have LVM PV/VG on that "mpath-resize-test" device.. Can that be the problem?

from /var/log/messages:

multipathd: mpath-resize-test: resize map (operator)
multipathd: mpath-resize-test: map is still the same size (8417280)

And now it seems that device is not working anymore.. "touch /mnt/testvol/test" hangs and doesn't do anything.. also "lvdisplay" just hangs. 

Weird.

Thanks Ben/Jeff.
Now it works perfect.

Ritesh: 

- How did you test it? 
- Filesystem directly on top of dm-multipath device? 
- Does "multipath -ll" show the new/bigger size for you after resize map? 

I'm using LVM PV/VG on top of dm-multipath device, and I can't get the dm-multipath device to grow on the fly.. My multipath-device consists of 2 paths (scsi devices) to iSCSI storage. 

Maybe I'll try again..

Thanks!

Pasi:

This package only does the userspace part of the fix.  I'm not sure when the kernel part was added, but you need a recent kernel, possibly a beta-release kernel.

You should be able to check if this works be doing a

# blockdev --getsize <device>

after you have resized your scsi devices.  If this shows the new size, then everything should work.  If not, then this isn't going to work without
a newer kernel or extra steps.

For growing a device, the extra steps involve running

# multipathd -k
> del path <path1_from_the_device>
> add path <path1_from_the_device>
> del path <path2_from_the_device>
> add path <path2_from_the_device>
...
for all the paths in the device. This allows the size to change. Then you 
can run
> resize map <device>
> CTRL-D

It's odd the the device is all locked up. Can you run
# dmsetup info <device>

The "State" should say "ACTIVE".  If is says "SUSPENDED", then that's the problem.  You can run

# dmsetup resume <device>

To get the device back to the active state (hopefully.  It might not work at all
without the kernel fix). In any case, let me know what you find.

I'm running patched kernel from http://people.redhat.com/jmoyer/dio/rhel5/ so that should be all fine. 

Like said, I was able to resize SCSI (iSCSI) devices on the fly. Kernel sees the new/grown size for SCSI devices. "blockdev --getsize" reports the new size, dmesg shows it and "/proc/partitions" shows the new size.

The problem is I didn't do that del/add path magic earlier. I just tried 'resize map' straight out.

"dmsetup info" says the device is SUSPENDED, but "dmsetup resume" doesn't help. it's still SUSPENDED.

I was able to del path from the mpath device, but now I can't add that path back:

/var/log/messages:
multipathd: mpath-resize-test: uev_add_path sleep
multipathd: mpath-resize-test: failed in domap for addition of new path sdc

And now for example "multipathd -k" just hangs and doesn't do anything.. 
"multipath -ll" works though. 

Any more ideas? :) 

Thanks!

Pasi,

I'm not sure how you're doing it. The only difference in my and your's test setup is LVM.

I've tested it on a multipathed device with a filesystem on top of it. Online resize works perfect.

Even for LVM on top of multipath, I think it should work perfect. You'll only have extend the VG/LV after your resize the multpathed device.

Ritesh: 

Thanks for the reply.

I know I need to pvresize the PV/VG after resizing multipath device. 

So far I was able to resize SCSI devices (that make up the multipath device) just fine online.

But the problem is I can't get the dm-multipath device to resize/grow.. removing a path was successful, but trying to add it back fails and hangs "multipathd -k".

I think I'll re-test again, first without LVM just with a filesystem on top of dm-multipath device, and if that works, then continue with LVM on top of dm-multipath.

Maybe there's something wrong with my multipath.conf causing the problem with resizing (adding paths back). At least the dm-multipath device has went into SUSPENDED mode and I can't get it back to ACTIVE.

Now I got it working! 

You can see the steps I did for successful online resize here:
http://pasik.reaktio.net/rhel5-online-iscsi-resize-test.txt

I tried with both ext3 filesystem directly on top of dm-mpath device, and LVM PV/VG on top of dm-mpath device. Both worked OK now.

Why it didn't work for the first time is still a bit of mystery.. I had to actually power cycle the whole server, because mpath device went into SUSPENDED mode and I couldn't get it back to ACTIVE.. and there was IO waiting for execution so things were just stalling and nothing happening.. 'reboot' command didn't work either.

I suspect the problem was that I didn't reload/restart multipathd to pick up the proper configuration for mpath-device before I started to experiment with online resizing.. 

Anyway, now it seems to work.. 

Thanks a lot!

Verified in RHEL5U3 snapshot3. I followed steps mentioned in comment #22, things seems to be working fine after deleting/adding all mapped paths to the concerned multipathed device and resizing that multipathed device.

Going forward we should work on script which will do all this stuff automatically.

Thank you.

Release note added. If any revisions are required, please set the 
"requires_release_notes" flag to "?" and edit the "Release Notes" field accordingly.
All revisions will be proofread by the Engineering Content Services team.

New Contents:
There is a new multipathd command, "resize map <mapname>". This command is used to cause dm-multipath to adjust to a change in the size of the underlying block storage device. 

Use the the following procedure, after you resize the underlying block device: 

# multipathd -k
multipathd> del path <path-name>
ok
multipathd> add path <path-name>
ok

<repeat for each path to the device>

multipathd> resize map <map name>

For example:

# multipath -ll

mpath-resize-test (36090a018c032e4801e8e341700008039) dm-5 EQLOGIC,100E-00
[size=2.0G][features=0][hwhandler=0][rw]
\_ round-robin 0 [prio=1][active]
 \_ 2:0:0:0 sdd 8:48  [active][ready]
\_ round-robin 0 [prio=1][enabled]
 \_ 3:0:0:0 sdc 8:32  [active][ready]


# multipathd -k
multipathd> del path sdc
ok
multipathd> add path sdc
ok
multipathd> del path sdd
ok
multipathd> add path sdd
ok
multipathd> resize map mpath-resize-test
ok

There is one restriction on the use of the device mapper multipath resize command. You can not resize a device-mapper device while there commands queued to that device. That is, do not use the resize command when no_path_retry is set to "queue", and there are no active paths to the device.

editted.  With 5.3 kernel, you don't need to remove and readd the paths anymore. You can just resize the LUN and then resize the multipath device.

Release note updated. If any revisions are required, please set the 
"requires_release_notes"  flag to "?" and edit the "Release Notes" field accordingly.
All revisions will be proofread by the Engineering Content Services team.

Diffed Contents:
@@ -1,39 +1,11 @@
 There is a new multipathd command, "resize map <mapname>". This command is used to cause dm-multipath to adjust to a change in the size of the underlying block storage device. 
 
-Use the the following procedure, after you resize the underlying block device: 
+After resizing the underlying block device, you can resize your multipath device by running:
 
-# multipathd -k
-multipathd> del path <path-name>
-ok
-multipathd> add path <path-name>
-ok
+# multipathd -k"resize map <mapname>"
 
-<repeat for each path to the device>
-
-multipathd> resize map <map name>
-
 For example:
 
-# multipath -ll
-
-mpath-resize-test (36090a018c032e4801e8e341700008039) dm-5 EQLOGIC,100E-00
-[size=2.0G][features=0][hwhandler=0][rw]
-\_ round-robin 0 [prio=1][active]
- \_ 2:0:0:0 sdd 8:48  [active][ready]
-\_ round-robin 0 [prio=1][enabled]
- \_ 3:0:0:0 sdc 8:32  [active][ready]
-
-
-# multipathd -k
-multipathd> del path sdc
-ok
-multipathd> add path sdc
-ok
-multipathd> del path sdd
-ok
-multipathd> add path sdd
-ok
-multipathd> resize map mpath-resize-test
-ok
+# multipathd -k"resize map mpath0"
 
 There is one restriction on the use of the device mapper multipath resize command. You can not resize a device-mapper device while there commands queued to that device. That is, do not use the resize command when no_path_retry is set to "queue", and there are no active paths to the device.

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHEA-2009-0232.html