This service will be undergoing maintenance at 00:00 UTC, 2016-08-01. It is expected to last about 1 hours
Bug 867593 - installer must stop relying on being able to tell kpartx to use a disk/partition delimiter
installer must stop relying on being able to tell kpartx to use a disk/partit...
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: anaconda (Show other bugs)
18
ppc64 Linux
unspecified Severity urgent
: ---
: ---
Assigned To: David Lehman
Fedora Extras Quality Assurance
AcceptedBlocker
:
Depends On:
Blocks: F18Beta/F18BetaBlocker F18Betappc
  Show dependency treegraph
 
Reported: 2012-10-17 15:48 EDT by Mark Hamzy
Modified: 2012-11-08 04:13 EST (History)
29 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-11-08 04:13:04 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
Reclaim Disk Space dialog (52.30 KB, image/png)
2012-10-17 15:48 EDT, Mark Hamzy
no flags Details
traceback generated with kill -USR2 `cat /var/run/anaconda.pid` (278.90 KB, text/plain)
2012-10-17 15:49 EDT, Mark Hamzy
no flags Details
anaconda.log (4.09 KB, application/octet-stream)
2012-10-17 15:50 EDT, Mark Hamzy
no flags Details
storage.log (151.63 KB, application/octet-stream)
2012-10-17 15:50 EDT, Mark Hamzy
no flags Details
patch (1.45 KB, patch)
2012-10-17 17:10 EDT, Brian Lane
no flags Details | Diff
traceback generated when patch was used (220.31 KB, text/plain)
2012-10-17 17:24 EDT, Mark Hamzy
no flags Details
Comment (97.30 KB, text/plain)
2012-10-24 13:44 EDT, Mark Hamzy
no flags Details

  None (edit)
Description Mark Hamzy 2012-10-17 15:48:41 EDT
Created attachment 629046 [details]
Reclaim Disk Space dialog

Description of problem:
When installing Fedora 18, I see no free space to reclaim on a multipath system with anaconda-18.17-1.fc18.
Comment 1 Mark Hamzy 2012-10-17 15:49:53 EDT
Created attachment 629047 [details]
traceback generated with kill -USR2 `cat /var/run/anaconda.pid`

traceback generated with kill -USR2 `cat /var/run/anaconda.pid`
Comment 2 Mark Hamzy 2012-10-17 15:50:14 EDT
Created attachment 629048 [details]
anaconda.log
Comment 3 Mark Hamzy 2012-10-17 15:50:35 EDT
Created attachment 629049 [details]
storage.log
Comment 4 Mark Hamzy 2012-10-17 16:36:45 EDT
[anaconda root@riley6 ~]# ls /dev/mapper/
control  live-rw  mpatha  mpathap1  mpathap2  mpathap3  mpathap4  mpathap5

[anaconda root@riley6 ~]# python
Python 2.7.3 (default, Aug 15 2012, 09:57:57)
[GCC 4.7.1 20120720 (Red Hat 4.7.1-5)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import parted
>>> disk = parted.Disk(device=parted.Device(path="/dev/mapper/mpatha"))
>>> rc = disk.getPartitionByPath("/dev/mapper/mpathap1")
>>> print rc
None
>>> rc = disk.getPartitionByPath("/dev/mapper/mpatha1")
>>> print rc
parted.Partition instance --
  disk: <parted.disk.Disk object at 0xfff95bc3110>  fileSystem: None
  number: 1  path: /dev/mapper/mpatha1  type: 0
  name: None  active: True  busy: False
  geometry: <parted.geometry.Geometry object at 0xfff95bc3310>  PedPartition: <_ped.Partition object at 0xfff9ca47290>
Comment 5 Mark Hamzy 2012-10-17 16:38:56 EDT
<dlehman> hamzy2: the simple test is to instantiate a parted.Disk for mpatha which has at least one partition and see if disk.getPartitionByPath("/dev/mapper/mpathap1") returns None.
<dlehman> it's weird, though, that this was never an issue before.
<hamzy2> parted.Disk(device=parted.Device(path="/dev/mapper/mpatha"))
<hamzy2> dlehman, like that ^^ ?
<dlehman> sure
<hamzy2> >>> rc = disk.getPartitionByPath("/dev/mapper/mpathap1")
<hamzy2> >>> print rc
<hamzy2> None
<hamzy2> dlehman, ^^
<dlehman> so there's your problem. anaconda runs kpartx with a delimiter of 'p' like it has for however many years and parted or pyparted has stopped using the delimiter and therefore would probably return a parted.Partition instance if you looked up "/dev/mapper/mpatha1" instead
<hamzy2> dlehman, disk.getPartitionByPath("/dev/mapper/mpatha1") works
<dlehman> so there you go. file a bug against pyparted. they can check and reassign to parted if needed.
Comment 6 Brian Lane 2012-10-17 16:57:01 EDT
parted behavior now matches kpartx, pX is only added if the device name ends in a digit. anaconda needs to stop passing -p p to kpartx.
Comment 7 Brian Lane 2012-10-17 17:10:43 EDT
Created attachment 629091 [details]
patch
Comment 8 Mark Hamzy 2012-10-17 17:24:14 EDT
Created attachment 629092 [details]
traceback generated when patch was used
Comment 9 Mark Hamzy 2012-10-19 10:53:37 EDT
We saw the following in syslog.  Need to investigate what is happening here...

[anaconda root@riley6 ~]# grep systemd-udevd /tmp/syslog 
14:37:00,065 ERR systemd-udevd: symlink '../../sda' '/dev/disk/by-id/scsi-3600507680281026d6800000000000001.udev-tmp' failed: File exists
14:37:00,066 ERR systemd-udevd: symlink '../../sdc' '/dev/disk/by-id/scsi-3600507680281026d6800000000000001.udev-tmp' failed: File exists
14:37:04,061 ERR systemd-udevd: rename '/dev/disk/by-uuid/3a8d8025-8d13-41ef-a2eb-46ffdc4b371a.udev-tmp' '/dev/disk/by-uuid/3a8d8025-8d13-41ef-a2eb-46ffdc4b371a' failed: No such file or directory
14:37:06,347 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap3' found, link to '/dev/dm-4' will not be created
14:37:06,347 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap2' found, link to '/dev/dm-3' will not be created
14:37:06,351 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap1' found, link to '/dev/dm-2' will not be created
14:37:06,352 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap5' found, link to '/dev/dm-6' will not be created
14:37:06,355 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap4' found, link to '/dev/dm-5' will not be created
14:37:06,436 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap3' found, link to '/dev/dm-4' will not be created
14:37:06,443 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap4' found, link to '/dev/dm-5' will not be created
14:37:06,453 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap4' found, link to '/dev/dm-5' will not be created
14:37:06,456 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap1' found, link to '/dev/dm-2' will not be created
14:37:06,459 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap2' found, link to '/dev/dm-3' will not be created
14:37:06,463 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap5' found, link to '/dev/dm-6' will not be created
14:37:06,463 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap3' found, link to '/dev/dm-4' will not be created
14:37:06,475 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap1' found, link to '/dev/dm-2' will not be created
14:37:06,482 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap2' found, link to '/dev/dm-3' will not be created
14:37:06,486 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap5' found, link to '/dev/dm-6' will not be created
14:37:06,608 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap4' found, link to '/dev/dm-5' will not be created
14:37:06,608 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap1' found, link to '/dev/dm-2' will not be created
14:37:06,615 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap5' found, link to '/dev/dm-6' will not be created
14:37:06,617 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap3' found, link to '/dev/dm-4' will not be created
14:37:06,618 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap2' found, link to '/dev/dm-3' will not be created
Comment 10 Brian Lane 2012-10-19 20:38:49 EDT
I think this is a systemd-udev problem. I can reproduce this on a f18 install with systemd-194-1.fc18.x86_64 installed.

Reproducer:

dd if=/dev/null of=disk.img bs=1M seek=1k
parted -s disk.img mklabel msdos
parted -s disk.img mkpart p 1M 10M
kpartx -a disk.img

syslog has this in it:
Oct 19 20:32:12 localhost systemd-udevd[2622]: conflicting device node '/dev/mapper/loop0p1' found, link to '/dev/dm-0' will not be created

It didn't symlink to ../dm-0 even though it exists.
brw-r--r--. 1 root root 253,   0 Oct 19 20:32 loop0p1

brw-rw----. 1 root disk 253, 0 Oct 19 20:32 /dev/dm-0
Comment 11 Kay Sievers 2012-10-22 06:38:19 EDT
Device-mapper seems to mknod() things in /dev, which just can not work
correctly today.

There is nothing udev can fix here, it will never touch any device
node, which should not exist in the first place, that is in the way.
Comment 12 Mark Hamzy 2012-10-24 13:44:39 EDT
Created attachment 915507 [details]
Comment

(This comment was longer than 65,535 characters and has been moved to an attachment by Red Hat Bugzilla).
Comment 13 Ben Marzinski 2012-10-24 15:09:34 EDT
The bug for making multipath and kpartx create links is 864368 but it depends on a libdevmapper fix.
Comment 14 Mark Hamzy 2012-10-25 11:45:07 EDT
I believe that the scratch build for lvm2 contains the necessary libdevmapper fix:

(PPC builds)
http://ppc.koji.fedoraproject.org/koji/taskinfo?taskID=763756

(x86_64.i686)
https://koji.fedoraproject.org/koji/taskinfo?taskID=4625132
Comment 15 Mark Hamzy 2012-10-26 15:01:38 EDT
I've built a test iso with

http://ppc.koji.fedoraproject.org/koji/buildinfo?buildID=116233
http://ppc.koji.fedoraproject.org/koji/taskinfo?taskID=763756

and https://bugzilla.redhat.com/show_bug.cgi?id=867593 still happens.

I've verified that libdevmapper is the same.
<mock-chroot>[root@bluebill tmp]# rpm2cpio /home/f18-private-packages/packages/device-mapper-libs-1.02.76-1.1.fc18.ppc64.rpm | cpio -id
676 blocks
<mock-chroot>[root@bluebill tmp]# sha256sum usr/lib64/libdevmapper.so.1.02 
0e3c006597f9c4ca20c247b682e6f16ed93fa8e383e8ecc2d14dcebad1ec11e9  usr/lib64/libdevmapper.so.1.02

and

[anaconda root@riley6 ~]# sha256sum /usr/lib64/libdevmapper.so.1.02             
0e3c006597f9c4ca20c247b682e6f16ed93fa8e383e8ecc2d14dcebad1ec11e9  /usr/lib64/libdevmapper.so.1.02

I see the following messages in /tmp/syslog:

17:48:29,107 ERR systemd-udevd: symlink '../../sdd' '/dev/disk/by-id/scsi-3600507680281026d6800000000000001.udev-tmp' failed: File exists
17:48:29,110 ERR systemd-udevd: symlink '../../sdb' '/dev/disk/by-id/wwn-0x600507680281026d6800000000000001.udev-tmp' failed: File exists
17:48:29,439 ERR systemd-udevd: rename '/dev/disk/by-id/scsi-3600507680281026d6800000000000001.udev-tmp' '/dev/disk/by-id/scsi-3600507680281026d6800000000000001' failed: No such file or directory
...
17:48:33,478 ERR systemd-udevd: symlink '../../sda' '/dev/disk/by-id/scsi-3600507680281026d6800000000000001.udev-tmp' failed: File exists
17:48:33,479 ERR systemd-udevd: symlink '../../sdc' '/dev/disk/by-id/wwn-0x600507680281026d6800000000000001.udev-tmp' failed: File exists

I applied the following anaconda patch:

[root@bluebill anaconda-18.19-1]# git diff
diff --git a/pyanaconda/storage/devices.py b/pyanaconda/storage/devices.py
index 1246177..b965e06 100644
--- a/pyanaconda/storage/devices.py
+++ b/pyanaconda/storage/devices.py
@@ -1752,7 +1752,7 @@ class DMDevice(StorageDevice):
     def setupPartitions(self):
         log_method_call(self, name=self.name, kids=self.kids)
         rc = iutil.execWithRedirect("kpartx",
-                                ["-a", "-p", "p", self.path],
+                                ["-a", self.path],
                                 stdout = "/dev/tty5",
                                 stderr = "/dev/tty5")
         if rc:
@@ -1762,7 +1762,7 @@ class DMDevice(StorageDevice):
     def teardownPartitions(self):
         log_method_call(self, name=self.name, kids=self.kids)
         rc = iutil.execWithRedirect("kpartx",
-                                ["-d", "-p", "p", self.path],
+                                ["-d", self.path],
                                 stdout = "/dev/tty5",
                                 stderr = "/dev/tty5")
         if rc:

And I get the following exception:

anaconda 18.19 exception report
Traceback (most recent call first):
  File "/tmp/updates/pyanaconda/storage/devices.py", line 1759, in setupPartitions
    raise DMError("partition activation failed for '%s'" % self.name)
  File "/tmp/updates/pyanaconda/storage/devices.py", line 3308, in _postSetup
    self.setupPartitions()
  File "/tmp/updates/pyanaconda/storage/devices.py", line 713, in setup
    self._postSetup()
  File "/tmp/updates/pyanaconda/storage/devicetree.py", line 1904, in _populate
    mp.setup()
  File "/tmp/updates/pyanaconda/storage/devicetree.py", line 1843, in populate
    self._populate()
  File "/tmp/updates/pyanaconda/storage/__init__.py", line 462, in reset
    self.devicetree.populate(cleanupOnly=cleanupOnly)
  File "/tmp/updates/pyanaconda/storage/__init__.py", line 104, in storageInitialize
    storage.reset()
  File "/usr/lib64/python2.7/threading.py", line 504, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/tmp/updates/pyanaconda/threads.py", line 91, in run
    threading.Thread.run(self, *args, **kwargs)
DMError: partition activation failed for 'mpatha'

This is what I see when I run the kpartx command on the bash prompt:

[anaconda root@riley6 ~]# kpartx -a /dev/mapper/mpatha
device-mapper: create ioctl on mpatha1 failed: Device or resource busy
create/reload failed on mpatha1
device-mapper: create ioctl on mpatha2 failed: Device or resource busy
create/reload failed on mpatha2
device-mapper: create ioctl on mpatha3 failed: Device or resource busy
create/reload failed on mpatha3
device-mapper: create ioctl on mpatha4 failed: Device or resource busy
create/reload failed on mpatha4
device-mapper: create ioctl on mpatha5 failed: Device or resource busy
create/reload failed on mpatha5

Is there more detailed debugging I can do for you?
Comment 16 Mark Hamzy 2012-10-29 14:59:51 EDT
<dlehman> I think you may want to reassign your bug back to anaconda. We need to stop passing '-p p', start passing '-s', and there is also work to do so other things work without the 'p' delimiter which we've always had.
Comment 17 David Lehman 2012-10-29 15:21:27 EDT
We won't be able to install to dmraid or multipath until this is worked out.
Comment 18 Adam Williamson 2012-10-29 15:39:41 EDT
Proposing as Beta blocker, then, FW raid is supposed to work at Beta:

"The installer must be able to create and install to software, hardware or BIOS RAID-0, RAID-1 or RAID-5 partitions for anything except /boot"
Comment 19 Adam Williamson 2012-10-31 12:31:39 EDT
Discussed at 2012-10-31 blocker review meeting: http://meetbot.fedoraproject.org/fedora-qa/2012-10-31/f18beta-blocker-review-6.2012-10-31-16.00.log.txt . Accepted as a blocker per criterion cited in comment #18.
Comment 20 Fedora Update System 2012-11-02 00:04:49 EDT
anaconda-18.23-1.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/anaconda-18.23-1.fc18
Comment 21 Fedora Update System 2012-11-02 14:39:21 EDT
Package anaconda-18.23-1.fc18:
* should fix your issue,
* was pushed to the Fedora 18 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing anaconda-18.23-1.fc18'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-17509/anaconda-18.23-1.fc18
then log in and leave karma (feedback).
Comment 22 Fedora Update System 2012-11-02 21:04:18 EDT
anaconda-18.24-1.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/anaconda-18.24-1.fc18
Comment 23 Fedora Update System 2012-11-03 15:29:02 EDT
Package anaconda-18.24-1.fc18:
* should fix your issue,
* was pushed to the Fedora 18 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing anaconda-18.24-1.fc18'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-17543/anaconda-18.24-1.fc18
then log in and leave karma (feedback).
Comment 24 Fedora Update System 2012-11-05 20:39:31 EST
anaconda-18.25-1.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/anaconda-18.25-1.fc18
Comment 25 Adam Williamson 2012-11-05 20:42:17 EST
Can someone please re-test and confirm the fix with Beta TC7? Thanks!
Comment 26 Mark Hamzy 2012-11-06 15:21:29 EST
Do you have an Intel multipath environment to test on, Adam? The ppc builds are currently broken.
Comment 27 Adam Williamson 2012-11-06 17:13:02 EST
No. tflink may have access to one.
Comment 28 Fedora Update System 2012-11-06 21:11:25 EST
anaconda-18.26-1.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/anaconda-18.26-1.fc18
Comment 29 Tim Flink 2012-11-07 17:10:20 EST
I did a default install on an i686 machine with a clean promise RAID1 (dmraid). Autopart selection went off without a hitch and the install got past the partitioning and packaging phases.

However, I hit #873463 before the install finished. Since this bug manifested before this point, I'm moving to VERIFIED.
Comment 30 Adam Williamson 2012-11-08 04:13:04 EST
18.26 went stable. Closing. (Bodhi closing of bugs when updates go stable is currently broken).

Note You need to log in before you can comment on or make changes to this bug.