Bug 867593 - installer must stop relying on being able to tell kpartx to use a disk/partition delimiter
Summary: installer must stop relying on being able to tell kpartx to use a disk/partit...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: anaconda
Version: 18
Hardware: ppc64
OS: Linux
unspecified
urgent
Target Milestone: ---
Assignee: David Lehman
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: AcceptedBlocker
Depends On:
Blocks: F18Beta, F18BetaBlocker F18Betappc
TreeView+ depends on / blocked
 
Reported: 2012-10-17 19:48 UTC by Mark Hamzy
Modified: 2012-11-08 09:13 UTC (History)
29 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2012-11-08 09:13:04 UTC


Attachments (Terms of Use)
Reclaim Disk Space dialog (52.30 KB, image/png)
2012-10-17 19:48 UTC, Mark Hamzy
no flags Details
traceback generated with kill -USR2 `cat /var/run/anaconda.pid` (278.90 KB, text/plain)
2012-10-17 19:49 UTC, Mark Hamzy
no flags Details
anaconda.log (4.09 KB, application/octet-stream)
2012-10-17 19:50 UTC, Mark Hamzy
no flags Details
storage.log (151.63 KB, application/octet-stream)
2012-10-17 19:50 UTC, Mark Hamzy
no flags Details
patch (1.45 KB, patch)
2012-10-17 21:10 UTC, Brian Lane
no flags Details | Diff
traceback generated when patch was used (220.31 KB, text/plain)
2012-10-17 21:24 UTC, Mark Hamzy
no flags Details
Comment (97.30 KB, text/plain)
2012-10-24 17:44 UTC, Mark Hamzy
no flags Details

Description Mark Hamzy 2012-10-17 19:48:41 UTC
Created attachment 629046 [details]
Reclaim Disk Space dialog

Description of problem:
When installing Fedora 18, I see no free space to reclaim on a multipath system with anaconda-18.17-1.fc18.

Comment 1 Mark Hamzy 2012-10-17 19:49:53 UTC
Created attachment 629047 [details]
traceback generated with kill -USR2 `cat /var/run/anaconda.pid`

traceback generated with kill -USR2 `cat /var/run/anaconda.pid`

Comment 2 Mark Hamzy 2012-10-17 19:50:14 UTC
Created attachment 629048 [details]
anaconda.log

Comment 3 Mark Hamzy 2012-10-17 19:50:35 UTC
Created attachment 629049 [details]
storage.log

Comment 4 Mark Hamzy 2012-10-17 20:36:45 UTC
[anaconda root@riley6 ~]# ls /dev/mapper/
control  live-rw  mpatha  mpathap1  mpathap2  mpathap3  mpathap4  mpathap5

[anaconda root@riley6 ~]# python
Python 2.7.3 (default, Aug 15 2012, 09:57:57)
[GCC 4.7.1 20120720 (Red Hat 4.7.1-5)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import parted
>>> disk = parted.Disk(device=parted.Device(path="/dev/mapper/mpatha"))
>>> rc = disk.getPartitionByPath("/dev/mapper/mpathap1")
>>> print rc
None
>>> rc = disk.getPartitionByPath("/dev/mapper/mpatha1")
>>> print rc
parted.Partition instance --
  disk: <parted.disk.Disk object at 0xfff95bc3110>  fileSystem: None
  number: 1  path: /dev/mapper/mpatha1  type: 0
  name: None  active: True  busy: False
  geometry: <parted.geometry.Geometry object at 0xfff95bc3310>  PedPartition: <_ped.Partition object at 0xfff9ca47290>

Comment 5 Mark Hamzy 2012-10-17 20:38:56 UTC
<dlehman> hamzy2: the simple test is to instantiate a parted.Disk for mpatha which has at least one partition and see if disk.getPartitionByPath("/dev/mapper/mpathap1") returns None.
<dlehman> it's weird, though, that this was never an issue before.
<hamzy2> parted.Disk(device=parted.Device(path="/dev/mapper/mpatha"))
<hamzy2> dlehman, like that ^^ ?
<dlehman> sure
<hamzy2> >>> rc = disk.getPartitionByPath("/dev/mapper/mpathap1")
<hamzy2> >>> print rc
<hamzy2> None
<hamzy2> dlehman, ^^
<dlehman> so there's your problem. anaconda runs kpartx with a delimiter of 'p' like it has for however many years and parted or pyparted has stopped using the delimiter and therefore would probably return a parted.Partition instance if you looked up "/dev/mapper/mpatha1" instead
<hamzy2> dlehman, disk.getPartitionByPath("/dev/mapper/mpatha1") works
<dlehman> so there you go. file a bug against pyparted. they can check and reassign to parted if needed.

Comment 6 Brian Lane 2012-10-17 20:57:01 UTC
parted behavior now matches kpartx, pX is only added if the device name ends in a digit. anaconda needs to stop passing -p p to kpartx.

Comment 7 Brian Lane 2012-10-17 21:10:43 UTC
Created attachment 629091 [details]
patch

Comment 8 Mark Hamzy 2012-10-17 21:24:14 UTC
Created attachment 629092 [details]
traceback generated when patch was used

Comment 9 Mark Hamzy 2012-10-19 14:53:37 UTC
We saw the following in syslog.  Need to investigate what is happening here...

[anaconda root@riley6 ~]# grep systemd-udevd /tmp/syslog 
14:37:00,065 ERR systemd-udevd: symlink '../../sda' '/dev/disk/by-id/scsi-3600507680281026d6800000000000001.udev-tmp' failed: File exists
14:37:00,066 ERR systemd-udevd: symlink '../../sdc' '/dev/disk/by-id/scsi-3600507680281026d6800000000000001.udev-tmp' failed: File exists
14:37:04,061 ERR systemd-udevd: rename '/dev/disk/by-uuid/3a8d8025-8d13-41ef-a2eb-46ffdc4b371a.udev-tmp' '/dev/disk/by-uuid/3a8d8025-8d13-41ef-a2eb-46ffdc4b371a' failed: No such file or directory
14:37:06,347 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap3' found, link to '/dev/dm-4' will not be created
14:37:06,347 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap2' found, link to '/dev/dm-3' will not be created
14:37:06,351 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap1' found, link to '/dev/dm-2' will not be created
14:37:06,352 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap5' found, link to '/dev/dm-6' will not be created
14:37:06,355 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap4' found, link to '/dev/dm-5' will not be created
14:37:06,436 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap3' found, link to '/dev/dm-4' will not be created
14:37:06,443 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap4' found, link to '/dev/dm-5' will not be created
14:37:06,453 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap4' found, link to '/dev/dm-5' will not be created
14:37:06,456 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap1' found, link to '/dev/dm-2' will not be created
14:37:06,459 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap2' found, link to '/dev/dm-3' will not be created
14:37:06,463 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap5' found, link to '/dev/dm-6' will not be created
14:37:06,463 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap3' found, link to '/dev/dm-4' will not be created
14:37:06,475 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap1' found, link to '/dev/dm-2' will not be created
14:37:06,482 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap2' found, link to '/dev/dm-3' will not be created
14:37:06,486 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap5' found, link to '/dev/dm-6' will not be created
14:37:06,608 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap4' found, link to '/dev/dm-5' will not be created
14:37:06,608 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap1' found, link to '/dev/dm-2' will not be created
14:37:06,615 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap5' found, link to '/dev/dm-6' will not be created
14:37:06,617 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap3' found, link to '/dev/dm-4' will not be created
14:37:06,618 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap2' found, link to '/dev/dm-3' will not be created

Comment 10 Brian Lane 2012-10-20 00:38:49 UTC
I think this is a systemd-udev problem. I can reproduce this on a f18 install with systemd-194-1.fc18.x86_64 installed.

Reproducer:

dd if=/dev/null of=disk.img bs=1M seek=1k
parted -s disk.img mklabel msdos
parted -s disk.img mkpart p 1M 10M
kpartx -a disk.img

syslog has this in it:
Oct 19 20:32:12 localhost systemd-udevd[2622]: conflicting device node '/dev/mapper/loop0p1' found, link to '/dev/dm-0' will not be created

It didn't symlink to ../dm-0 even though it exists.
brw-r--r--. 1 root root 253,   0 Oct 19 20:32 loop0p1

brw-rw----. 1 root disk 253, 0 Oct 19 20:32 /dev/dm-0

Comment 11 Kay Sievers 2012-10-22 10:38:19 UTC
Device-mapper seems to mknod() things in /dev, which just can not work
correctly today.

There is nothing udev can fix here, it will never touch any device
node, which should not exist in the first place, that is in the way.

Comment 12 Mark Hamzy 2012-10-24 17:44:39 UTC
Created attachment 915507 [details]
Comment

(This comment was longer than 65,535 characters and has been moved to an attachment by Red Hat Bugzilla).

Comment 13 Ben Marzinski 2012-10-24 19:09:34 UTC
The bug for making multipath and kpartx create links is 864368 but it depends on a libdevmapper fix.

Comment 14 Mark Hamzy 2012-10-25 15:45:07 UTC
I believe that the scratch build for lvm2 contains the necessary libdevmapper fix:

(PPC builds)
http://ppc.koji.fedoraproject.org/koji/taskinfo?taskID=763756

(x86_64.i686)
https://koji.fedoraproject.org/koji/taskinfo?taskID=4625132

Comment 15 Mark Hamzy 2012-10-26 19:01:38 UTC
I've built a test iso with

http://ppc.koji.fedoraproject.org/koji/buildinfo?buildID=116233
http://ppc.koji.fedoraproject.org/koji/taskinfo?taskID=763756

and https://bugzilla.redhat.com/show_bug.cgi?id=867593 still happens.

I've verified that libdevmapper is the same.
<mock-chroot>[root@bluebill tmp]# rpm2cpio /home/f18-private-packages/packages/device-mapper-libs-1.02.76-1.1.fc18.ppc64.rpm | cpio -id
676 blocks
<mock-chroot>[root@bluebill tmp]# sha256sum usr/lib64/libdevmapper.so.1.02 
0e3c006597f9c4ca20c247b682e6f16ed93fa8e383e8ecc2d14dcebad1ec11e9  usr/lib64/libdevmapper.so.1.02

and

[anaconda root@riley6 ~]# sha256sum /usr/lib64/libdevmapper.so.1.02             
0e3c006597f9c4ca20c247b682e6f16ed93fa8e383e8ecc2d14dcebad1ec11e9  /usr/lib64/libdevmapper.so.1.02

I see the following messages in /tmp/syslog:

17:48:29,107 ERR systemd-udevd: symlink '../../sdd' '/dev/disk/by-id/scsi-3600507680281026d6800000000000001.udev-tmp' failed: File exists
17:48:29,110 ERR systemd-udevd: symlink '../../sdb' '/dev/disk/by-id/wwn-0x600507680281026d6800000000000001.udev-tmp' failed: File exists
17:48:29,439 ERR systemd-udevd: rename '/dev/disk/by-id/scsi-3600507680281026d6800000000000001.udev-tmp' '/dev/disk/by-id/scsi-3600507680281026d6800000000000001' failed: No such file or directory
...
17:48:33,478 ERR systemd-udevd: symlink '../../sda' '/dev/disk/by-id/scsi-3600507680281026d6800000000000001.udev-tmp' failed: File exists
17:48:33,479 ERR systemd-udevd: symlink '../../sdc' '/dev/disk/by-id/wwn-0x600507680281026d6800000000000001.udev-tmp' failed: File exists

I applied the following anaconda patch:

[root@bluebill anaconda-18.19-1]# git diff
diff --git a/pyanaconda/storage/devices.py b/pyanaconda/storage/devices.py
index 1246177..b965e06 100644
--- a/pyanaconda/storage/devices.py
+++ b/pyanaconda/storage/devices.py
@@ -1752,7 +1752,7 @@ class DMDevice(StorageDevice):
     def setupPartitions(self):
         log_method_call(self, name=self.name, kids=self.kids)
         rc = iutil.execWithRedirect("kpartx",
-                                ["-a", "-p", "p", self.path],
+                                ["-a", self.path],
                                 stdout = "/dev/tty5",
                                 stderr = "/dev/tty5")
         if rc:
@@ -1762,7 +1762,7 @@ class DMDevice(StorageDevice):
     def teardownPartitions(self):
         log_method_call(self, name=self.name, kids=self.kids)
         rc = iutil.execWithRedirect("kpartx",
-                                ["-d", "-p", "p", self.path],
+                                ["-d", self.path],
                                 stdout = "/dev/tty5",
                                 stderr = "/dev/tty5")
         if rc:

And I get the following exception:

anaconda 18.19 exception report
Traceback (most recent call first):
  File "/tmp/updates/pyanaconda/storage/devices.py", line 1759, in setupPartitions
    raise DMError("partition activation failed for '%s'" % self.name)
  File "/tmp/updates/pyanaconda/storage/devices.py", line 3308, in _postSetup
    self.setupPartitions()
  File "/tmp/updates/pyanaconda/storage/devices.py", line 713, in setup
    self._postSetup()
  File "/tmp/updates/pyanaconda/storage/devicetree.py", line 1904, in _populate
    mp.setup()
  File "/tmp/updates/pyanaconda/storage/devicetree.py", line 1843, in populate
    self._populate()
  File "/tmp/updates/pyanaconda/storage/__init__.py", line 462, in reset
    self.devicetree.populate(cleanupOnly=cleanupOnly)
  File "/tmp/updates/pyanaconda/storage/__init__.py", line 104, in storageInitialize
    storage.reset()
  File "/usr/lib64/python2.7/threading.py", line 504, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/tmp/updates/pyanaconda/threads.py", line 91, in run
    threading.Thread.run(self, *args, **kwargs)
DMError: partition activation failed for 'mpatha'

This is what I see when I run the kpartx command on the bash prompt:

[anaconda root@riley6 ~]# kpartx -a /dev/mapper/mpatha
device-mapper: create ioctl on mpatha1 failed: Device or resource busy
create/reload failed on mpatha1
device-mapper: create ioctl on mpatha2 failed: Device or resource busy
create/reload failed on mpatha2
device-mapper: create ioctl on mpatha3 failed: Device or resource busy
create/reload failed on mpatha3
device-mapper: create ioctl on mpatha4 failed: Device or resource busy
create/reload failed on mpatha4
device-mapper: create ioctl on mpatha5 failed: Device or resource busy
create/reload failed on mpatha5

Is there more detailed debugging I can do for you?

Comment 16 Mark Hamzy 2012-10-29 18:59:51 UTC
<dlehman> I think you may want to reassign your bug back to anaconda. We need to stop passing '-p p', start passing '-s', and there is also work to do so other things work without the 'p' delimiter which we've always had.

Comment 17 David Lehman 2012-10-29 19:21:27 UTC
We won't be able to install to dmraid or multipath until this is worked out.

Comment 18 Adam Williamson 2012-10-29 19:39:41 UTC
Proposing as Beta blocker, then, FW raid is supposed to work at Beta:

"The installer must be able to create and install to software, hardware or BIOS RAID-0, RAID-1 or RAID-5 partitions for anything except /boot"

Comment 19 Adam Williamson 2012-10-31 16:31:39 UTC
Discussed at 2012-10-31 blocker review meeting: http://meetbot.fedoraproject.org/fedora-qa/2012-10-31/f18beta-blocker-review-6.2012-10-31-16.00.log.txt . Accepted as a blocker per criterion cited in comment #18.

Comment 20 Fedora Update System 2012-11-02 04:04:49 UTC
anaconda-18.23-1.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/anaconda-18.23-1.fc18

Comment 21 Fedora Update System 2012-11-02 18:39:21 UTC
Package anaconda-18.23-1.fc18:
* should fix your issue,
* was pushed to the Fedora 18 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing anaconda-18.23-1.fc18'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-17509/anaconda-18.23-1.fc18
then log in and leave karma (feedback).

Comment 22 Fedora Update System 2012-11-03 01:04:18 UTC
anaconda-18.24-1.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/anaconda-18.24-1.fc18

Comment 23 Fedora Update System 2012-11-03 19:29:02 UTC
Package anaconda-18.24-1.fc18:
* should fix your issue,
* was pushed to the Fedora 18 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing anaconda-18.24-1.fc18'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-17543/anaconda-18.24-1.fc18
then log in and leave karma (feedback).

Comment 24 Fedora Update System 2012-11-06 01:39:31 UTC
anaconda-18.25-1.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/anaconda-18.25-1.fc18

Comment 25 Adam Williamson 2012-11-06 01:42:17 UTC
Can someone please re-test and confirm the fix with Beta TC7? Thanks!

Comment 26 Mark Hamzy 2012-11-06 20:21:29 UTC
Do you have an Intel multipath environment to test on, Adam? The ppc builds are currently broken.

Comment 27 Adam Williamson 2012-11-06 22:13:02 UTC
No. tflink may have access to one.

Comment 28 Fedora Update System 2012-11-07 02:11:25 UTC
anaconda-18.26-1.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/anaconda-18.26-1.fc18

Comment 29 Tim Flink 2012-11-07 22:10:20 UTC
I did a default install on an i686 machine with a clean promise RAID1 (dmraid). Autopart selection went off without a hitch and the install got past the partitioning and packaging phases.

However, I hit #873463 before the install finished. Since this bug manifested before this point, I'm moving to VERIFIED.

Comment 30 Adam Williamson 2012-11-08 09:13:04 UTC
18.26 went stable. Closing. (Bodhi closing of bugs when updates go stable is currently broken).


Note You need to log in before you can comment on or make changes to this bug.