Bug 1254692

Summary: kpartx on multipath device returning before creation of dm partition device
Product: Red Hat Enterprise Linux 6 Reporter: John Pittman <jpittman>
Component: device-mapper-multipathAssignee: Ben Marzinski <bmarzins>
Status: CLOSED ERRATA QA Contact: Lin Li <lilin>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.7CC: agk, bmarzins, dwysocha, heinzm, lilin, msnitzer, prajnoha, prockai, rbalakri, zkabelac
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: device-mapper-multipath-0.4.9-88.el6 Doc Type: Bug Fix
Doc Text:
Cause: kpartx was by default returning without waiting for devices to be created. Consequence: Users expecting the devices to exist immediately after kpartx returns were being confused Fix: kpartx now by default waits until the devices are created before returning. Result: Users are no longer surprised if they expect the devivces to exist immediately after kpartx returns.
Story Points: ---
Clone Of:
: 1299648 (view as bug list) Environment:
Last Closed: 2016-05-10 20:12:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1172231, 1268411, 1299648    

Description John Pittman 2015-08-18 16:37:04 UTC
Description of problem:

kpartx run on a multipath device is creating the partition device mapper device, but it is returning before doing so.  This can cause various races.

The below script can reproduce:

    #!/bin/ksh
    i=/var/tmp/imgtst.img
    dd if=/dev/zero of=${i} bs=1024M count=2
    sfdisk -L ${i} << EOF
    0 -1
    EOF
    l=`losetup --find --show --read-only ${i}`;
    p=/dev/mapper/${l#/dev/}p1;
    kpartx -a -f -v "${l}";
    f=0; while [ $f -eq 0 ]; do
      test -e ${p} && f=1 || echo "NO: $(date '+%N')";
    done;
    echo "OK: $(date '+%N')";
    kpartx -d ${l};
    losetup -d ${l}
    rm -f ${i}

Version-Release number of selected component (if applicable):

kpartx-0.4.9-87.el6.x86_64

Issue does not occur at kpartx-0.4.9-46.el6.x86_64

Execute script mentioned to reproduce
Local recreation can be found at: http://pastebin.test.redhat.com/306295

Actual results:

kpartx returns before device mapper partition device is created

Expected results:

device mapper partition device should be created prior to the kpartx command returning

Comment 1 Peter Rajnoha 2015-08-19 07:28:16 UTC
(In reply to John Pittman from comment #0)
> device mapper partition device should be created prior to the kpartx command
> returning

Try kpartx -s ("-s     Sync mode. Don't return until the partitions are created").

Comment 2 John Pittman 2015-08-19 13:26:28 UTC
Thanks Peter! That worked in my environment.

Shouldn't this be the default behavior though?  Wouldn't we end up having issues with the multipath udev rules in /lib?

Thanks again.

John

Comment 3 Peter Rajnoha 2015-08-19 13:44:04 UTC
(In reply to John Pittman from comment #2)
> Thanks Peter! That worked in my environment.
> 
> Shouldn't this be the default behavior though?  Wouldn't we end up having
> issues with the multipath udev rules in /lib?

Yes, it would probably be better if the "udev-synced" mode is used by default and there's "--nosync" option instead to turn it off (for example if kpartx is called within udev rules where it's of little use).

I think it was done this way because it was probably expected that kpartx was not used that often directly on command line and its main use is for it to be run automatically from udev rule. Also, the udev-sync support for multipath tools (including kpartx) was added a bit later when the support was introduced in libdevmapper - so the reason may just be historical.

But yes, it would make more sense to enable this sync mode now by default and disable it on demand only - introducing a new --nosync option.

Ben, what do you think?

Comment 4 Ben Marzinski 2015-08-19 20:35:41 UTC
Sure. I don't see a need for kpartx to sync when udev runs it, but I have no problem switching the default behavior for when it is called manually.

Comment 7 Lin Li 2016-02-15 07:22:50 UTC
Reproduced on device-mapper-multipath-0.4.9-87.el6
1). # rpm -qa | grep kpartx
   kpartx-0.4.9-87.el6.x86_64

2). Execute script mentioned to reproduce
2+0 records in
2+0 records out
2147483648 bytes (2.1 GB) copied, 20.3739 s, 105 MB/s
Warning: /var/tmp/imgtst.img is not a block device
Disk /var/tmp/imgtst.img: cannot get geometry

Disk /var/tmp/imgtst.img: 261 cylinders, 255 heads, 63 sectors/track
 /var/tmp/imgtst.img: unrecognized partition table type
Old situation:
No partitions found
New situation:
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
/var/tmp/imgtst.img1          0+    259     260-   2088449+  83  Linux
/var/tmp/imgtst.img2          0       -       0          0    0  Empty
/var/tmp/imgtst.img3          0       -       0          0    0  Empty
/var/tmp/imgtst.img4          0       -       0          0    0  Empty
Warning: no primary partition is marked bootable (active)
This does not matter for LILO, but the DOS MBR will not boot this disk.
Successfully wrote the new partition table

Re-reading the partition table ...
BLKRRPART: Inappropriate ioctl for device

If you created or changed a DOS partition, /dev/foo7, say, then use dd(1)
to zero the first 512 bytes:  dd if=/dev/zero of=/dev/foo7 bs=512 count=1
(See fdisk(8).)
add map loop1p1 (253:8): 0 4176899 linear /dev/loop1 1
NO: 248214729
NO: 249224585
NO: 250160954
NO: 251099195
NO: 252138231
NO: 253058788
NO: 253938162
NO: 254594726
NO: 255435612
NO: 256228676
NO: 257079499
NO: 257996269
NO: 258687376
NO: 259417639
NO: 260207811
NO: 261062743
NO: 262027848
NO: 262957850
NO: 263839287
NO: 264742103
NO: 265644121
NO: 266535010
NO: 267479080
NO: 268395273
NO: 269337313
NO: 270245476
NO: 271203651
NO: 272173238
NO: 273075497
NO: 273890758
NO: 274679988
NO: 275601838
NO: 276450097
NO: 277306337
NO: 278207384
NO: 279140408
NO: 279984252
NO: 280764921
NO: 281736484
NO: 282612556
NO: 283524959
NO: 284398252
NO: 287282416
NO: 288089745
NO: 288887771
NO: 289664757
NO: 290559473
NO: 291433913
NO: 292287092
NO: 293332065
NO: 294304330
NO: 295282446
NO: 296213378
NO: 297178335
NO: 298054652
NO: 300324636
NO: 301278728
NO: 302200148
NO: 303174017
NO: 304061679
NO: 304960274
NO: 305823770
NO: 306708222
NO: 307634292
NO: 308505685
NO: 309408064
NO: 310275431
NO: 311235053
NO: 312046576
NO: 312739041
NO: 313364445
NO: 313967423
NO: 314760977
NO: 316805231
OK: 317768303

 

Verified on device-mapper-multipath-0.4.9-90.el6
1). rpm -qa | grep kpartx
    kpartx-0.4.9-90.el6.x86_64

2). Execute script mentioned
2+0 records in
2+0 records out
2147483648 bytes (2.1 GB) copied, 8.35367 s, 257 MB/s
Warning: /var/tmp/imgtst.img is not a block device
Disk /var/tmp/imgtst.img: cannot get geometry

Disk /var/tmp/imgtst.img: 261 cylinders, 255 heads, 63 sectors/track
 /var/tmp/imgtst.img: unrecognized partition table type
Old situation:
No partitions found
New situation:
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
/var/tmp/imgtst.img1          0+    259     260-   2088449+  83  Linux
/var/tmp/imgtst.img2          0       -       0          0    0  Empty
/var/tmp/imgtst.img3          0       -       0          0    0  Empty
/var/tmp/imgtst.img4          0       -       0          0    0  Empty
Warning: no primary partition is marked bootable (active)
This does not matter for LILO, but the DOS MBR will not boot this disk.
Successfully wrote the new partition table

Re-reading the partition table ...
BLKRRPART: Inappropriate ioctl for device

If you created or changed a DOS partition, /dev/foo7, say, then use dd(1)
to zero the first 512 bytes:  dd if=/dev/zero of=/dev/foo7 bs=512 count=1
(See fdisk(8).)
add map loop1p1 (253:8): 0 4176899 linear /dev/loop1 1
OK: 050449684



Test result:
device mapper partition device is created prior to the kpartx command returning

Comment 8 Lin Li 2016-02-15 07:23:31 UTC
change to verified.

Comment 10 errata-xmlrpc 2016-05-10 20:12:59 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-0777.html