Bug 867593
| Summary: | installer must stop relying on being able to tell kpartx to use a disk/partition delimiter | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | Mark Hamzy <hamzy> | ||||||||||||||||
| Component: | anaconda | Assignee: | David Lehman <dlehman> | ||||||||||||||||
| Status: | CLOSED ERRATA | QA Contact: | Fedora Extras Quality Assurance <extras-qa> | ||||||||||||||||
| Severity: | urgent | Docs Contact: | |||||||||||||||||
| Priority: | unspecified | ||||||||||||||||||
| Version: | 18 | CC: | agk, anaconda-maint-list, awilliam, bcl, bmarzins, bmr, dcantrell, dwysocha, extras-orphan, g.kaviyarasu, heinzm, johannbg, jonathan, lnykryn, lvm-team, mbroz, metherid, msekleta, msnitzer, notting, plautrba, prajnoha, prockai, robatino, systemd-maint, tflink, vanmeeuwen+fedora, vpavlin, zkabelac | ||||||||||||||||
| Target Milestone: | --- | ||||||||||||||||||
| Target Release: | --- | ||||||||||||||||||
| Hardware: | ppc64 | ||||||||||||||||||
| OS: | Linux | ||||||||||||||||||
| Whiteboard: | AcceptedBlocker | ||||||||||||||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||||||||||||||
| Doc Text: | Story Points: | --- | |||||||||||||||||
| Clone Of: | Environment: | ||||||||||||||||||
| Last Closed: | 2012-11-08 09:13:04 UTC | Type: | Bug | ||||||||||||||||
| Regression: | --- | Mount Type: | --- | ||||||||||||||||
| Documentation: | --- | CRM: | |||||||||||||||||
| Verified Versions: | Category: | --- | |||||||||||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||||||||||
| Embargoed: | |||||||||||||||||||
| Bug Depends On: | |||||||||||||||||||
| Bug Blocks: | 752660, 846990 | ||||||||||||||||||
| Attachments: |
|
||||||||||||||||||
Created attachment 629047 [details]
traceback generated with kill -USR2 `cat /var/run/anaconda.pid`
traceback generated with kill -USR2 `cat /var/run/anaconda.pid`
Created attachment 629048 [details]
anaconda.log
Created attachment 629049 [details]
storage.log
[anaconda root@riley6 ~]# ls /dev/mapper/ control live-rw mpatha mpathap1 mpathap2 mpathap3 mpathap4 mpathap5 [anaconda root@riley6 ~]# python Python 2.7.3 (default, Aug 15 2012, 09:57:57) [GCC 4.7.1 20120720 (Red Hat 4.7.1-5)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import parted >>> disk = parted.Disk(device=parted.Device(path="/dev/mapper/mpatha")) >>> rc = disk.getPartitionByPath("/dev/mapper/mpathap1") >>> print rc None >>> rc = disk.getPartitionByPath("/dev/mapper/mpatha1") >>> print rc parted.Partition instance -- disk: <parted.disk.Disk object at 0xfff95bc3110> fileSystem: None number: 1 path: /dev/mapper/mpatha1 type: 0 name: None active: True busy: False geometry: <parted.geometry.Geometry object at 0xfff95bc3310> PedPartition: <_ped.Partition object at 0xfff9ca47290> <dlehman> hamzy2: the simple test is to instantiate a parted.Disk for mpatha which has at least one partition and see if disk.getPartitionByPath("/dev/mapper/mpathap1") returns None.
<dlehman> it's weird, though, that this was never an issue before.
<hamzy2> parted.Disk(device=parted.Device(path="/dev/mapper/mpatha"))
<hamzy2> dlehman, like that ^^ ?
<dlehman> sure
<hamzy2> >>> rc = disk.getPartitionByPath("/dev/mapper/mpathap1")
<hamzy2> >>> print rc
<hamzy2> None
<hamzy2> dlehman, ^^
<dlehman> so there's your problem. anaconda runs kpartx with a delimiter of 'p' like it has for however many years and parted or pyparted has stopped using the delimiter and therefore would probably return a parted.Partition instance if you looked up "/dev/mapper/mpatha1" instead
<hamzy2> dlehman, disk.getPartitionByPath("/dev/mapper/mpatha1") works
<dlehman> so there you go. file a bug against pyparted. they can check and reassign to parted if needed.
parted behavior now matches kpartx, pX is only added if the device name ends in a digit. anaconda needs to stop passing -p p to kpartx. Created attachment 629091 [details]
patch
Created attachment 629092 [details]
traceback generated when patch was used
We saw the following in syslog. Need to investigate what is happening here... [anaconda root@riley6 ~]# grep systemd-udevd /tmp/syslog 14:37:00,065 ERR systemd-udevd: symlink '../../sda' '/dev/disk/by-id/scsi-3600507680281026d6800000000000001.udev-tmp' failed: File exists 14:37:00,066 ERR systemd-udevd: symlink '../../sdc' '/dev/disk/by-id/scsi-3600507680281026d6800000000000001.udev-tmp' failed: File exists 14:37:04,061 ERR systemd-udevd: rename '/dev/disk/by-uuid/3a8d8025-8d13-41ef-a2eb-46ffdc4b371a.udev-tmp' '/dev/disk/by-uuid/3a8d8025-8d13-41ef-a2eb-46ffdc4b371a' failed: No such file or directory 14:37:06,347 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap3' found, link to '/dev/dm-4' will not be created 14:37:06,347 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap2' found, link to '/dev/dm-3' will not be created 14:37:06,351 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap1' found, link to '/dev/dm-2' will not be created 14:37:06,352 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap5' found, link to '/dev/dm-6' will not be created 14:37:06,355 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap4' found, link to '/dev/dm-5' will not be created 14:37:06,436 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap3' found, link to '/dev/dm-4' will not be created 14:37:06,443 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap4' found, link to '/dev/dm-5' will not be created 14:37:06,453 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap4' found, link to '/dev/dm-5' will not be created 14:37:06,456 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap1' found, link to '/dev/dm-2' will not be created 14:37:06,459 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap2' found, link to '/dev/dm-3' will not be created 14:37:06,463 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap5' found, link to '/dev/dm-6' will not be created 14:37:06,463 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap3' found, link to '/dev/dm-4' will not be created 14:37:06,475 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap1' found, link to '/dev/dm-2' will not be created 14:37:06,482 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap2' found, link to '/dev/dm-3' will not be created 14:37:06,486 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap5' found, link to '/dev/dm-6' will not be created 14:37:06,608 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap4' found, link to '/dev/dm-5' will not be created 14:37:06,608 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap1' found, link to '/dev/dm-2' will not be created 14:37:06,615 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap5' found, link to '/dev/dm-6' will not be created 14:37:06,617 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap3' found, link to '/dev/dm-4' will not be created 14:37:06,618 ERR systemd-udevd: conflicting device node '/dev/mapper/mpathap2' found, link to '/dev/dm-3' will not be created I think this is a systemd-udev problem. I can reproduce this on a f18 install with systemd-194-1.fc18.x86_64 installed. Reproducer: dd if=/dev/null of=disk.img bs=1M seek=1k parted -s disk.img mklabel msdos parted -s disk.img mkpart p 1M 10M kpartx -a disk.img syslog has this in it: Oct 19 20:32:12 localhost systemd-udevd[2622]: conflicting device node '/dev/mapper/loop0p1' found, link to '/dev/dm-0' will not be created It didn't symlink to ../dm-0 even though it exists. brw-r--r--. 1 root root 253, 0 Oct 19 20:32 loop0p1 brw-rw----. 1 root disk 253, 0 Oct 19 20:32 /dev/dm-0 Device-mapper seems to mknod() things in /dev, which just can not work correctly today. There is nothing udev can fix here, it will never touch any device node, which should not exist in the first place, that is in the way. Created attachment 915507 [details]
Comment
(This comment was longer than 65,535 characters and has been moved to an attachment by Red Hat Bugzilla).
The bug for making multipath and kpartx create links is 864368 but it depends on a libdevmapper fix. I believe that the scratch build for lvm2 contains the necessary libdevmapper fix: (PPC builds) http://ppc.koji.fedoraproject.org/koji/taskinfo?taskID=763756 (x86_64.i686) https://koji.fedoraproject.org/koji/taskinfo?taskID=4625132 I've built a test iso with http://ppc.koji.fedoraproject.org/koji/buildinfo?buildID=116233 http://ppc.koji.fedoraproject.org/koji/taskinfo?taskID=763756 and https://bugzilla.redhat.com/show_bug.cgi?id=867593 still happens. I've verified that libdevmapper is the same. <mock-chroot>[root@bluebill tmp]# rpm2cpio /home/f18-private-packages/packages/device-mapper-libs-1.02.76-1.1.fc18.ppc64.rpm | cpio -id 676 blocks <mock-chroot>[root@bluebill tmp]# sha256sum usr/lib64/libdevmapper.so.1.02 0e3c006597f9c4ca20c247b682e6f16ed93fa8e383e8ecc2d14dcebad1ec11e9 usr/lib64/libdevmapper.so.1.02 and [anaconda root@riley6 ~]# sha256sum /usr/lib64/libdevmapper.so.1.02 0e3c006597f9c4ca20c247b682e6f16ed93fa8e383e8ecc2d14dcebad1ec11e9 /usr/lib64/libdevmapper.so.1.02 I see the following messages in /tmp/syslog: 17:48:29,107 ERR systemd-udevd: symlink '../../sdd' '/dev/disk/by-id/scsi-3600507680281026d6800000000000001.udev-tmp' failed: File exists 17:48:29,110 ERR systemd-udevd: symlink '../../sdb' '/dev/disk/by-id/wwn-0x600507680281026d6800000000000001.udev-tmp' failed: File exists 17:48:29,439 ERR systemd-udevd: rename '/dev/disk/by-id/scsi-3600507680281026d6800000000000001.udev-tmp' '/dev/disk/by-id/scsi-3600507680281026d6800000000000001' failed: No such file or directory ... 17:48:33,478 ERR systemd-udevd: symlink '../../sda' '/dev/disk/by-id/scsi-3600507680281026d6800000000000001.udev-tmp' failed: File exists 17:48:33,479 ERR systemd-udevd: symlink '../../sdc' '/dev/disk/by-id/wwn-0x600507680281026d6800000000000001.udev-tmp' failed: File exists I applied the following anaconda patch: [root@bluebill anaconda-18.19-1]# git diff diff --git a/pyanaconda/storage/devices.py b/pyanaconda/storage/devices.py index 1246177..b965e06 100644 --- a/pyanaconda/storage/devices.py +++ b/pyanaconda/storage/devices.py @@ -1752,7 +1752,7 @@ class DMDevice(StorageDevice): def setupPartitions(self): log_method_call(self, name=self.name, kids=self.kids) rc = iutil.execWithRedirect("kpartx", - ["-a", "-p", "p", self.path], + ["-a", self.path], stdout = "/dev/tty5", stderr = "/dev/tty5") if rc: @@ -1762,7 +1762,7 @@ class DMDevice(StorageDevice): def teardownPartitions(self): log_method_call(self, name=self.name, kids=self.kids) rc = iutil.execWithRedirect("kpartx", - ["-d", "-p", "p", self.path], + ["-d", self.path], stdout = "/dev/tty5", stderr = "/dev/tty5") if rc: And I get the following exception: anaconda 18.19 exception report Traceback (most recent call first): File "/tmp/updates/pyanaconda/storage/devices.py", line 1759, in setupPartitions raise DMError("partition activation failed for '%s'" % self.name) File "/tmp/updates/pyanaconda/storage/devices.py", line 3308, in _postSetup self.setupPartitions() File "/tmp/updates/pyanaconda/storage/devices.py", line 713, in setup self._postSetup() File "/tmp/updates/pyanaconda/storage/devicetree.py", line 1904, in _populate mp.setup() File "/tmp/updates/pyanaconda/storage/devicetree.py", line 1843, in populate self._populate() File "/tmp/updates/pyanaconda/storage/__init__.py", line 462, in reset self.devicetree.populate(cleanupOnly=cleanupOnly) File "/tmp/updates/pyanaconda/storage/__init__.py", line 104, in storageInitialize storage.reset() File "/usr/lib64/python2.7/threading.py", line 504, in run self.__target(*self.__args, **self.__kwargs) File "/tmp/updates/pyanaconda/threads.py", line 91, in run threading.Thread.run(self, *args, **kwargs) DMError: partition activation failed for 'mpatha' This is what I see when I run the kpartx command on the bash prompt: [anaconda root@riley6 ~]# kpartx -a /dev/mapper/mpatha device-mapper: create ioctl on mpatha1 failed: Device or resource busy create/reload failed on mpatha1 device-mapper: create ioctl on mpatha2 failed: Device or resource busy create/reload failed on mpatha2 device-mapper: create ioctl on mpatha3 failed: Device or resource busy create/reload failed on mpatha3 device-mapper: create ioctl on mpatha4 failed: Device or resource busy create/reload failed on mpatha4 device-mapper: create ioctl on mpatha5 failed: Device or resource busy create/reload failed on mpatha5 Is there more detailed debugging I can do for you? <dlehman> I think you may want to reassign your bug back to anaconda. We need to stop passing '-p p', start passing '-s', and there is also work to do so other things work without the 'p' delimiter which we've always had. We won't be able to install to dmraid or multipath until this is worked out. Proposing as Beta blocker, then, FW raid is supposed to work at Beta: "The installer must be able to create and install to software, hardware or BIOS RAID-0, RAID-1 or RAID-5 partitions for anything except /boot" Discussed at 2012-10-31 blocker review meeting: http://meetbot.fedoraproject.org/fedora-qa/2012-10-31/f18beta-blocker-review-6.2012-10-31-16.00.log.txt . Accepted as a blocker per criterion cited in comment #18. anaconda-18.23-1.fc18 has been submitted as an update for Fedora 18. https://admin.fedoraproject.org/updates/anaconda-18.23-1.fc18 Package anaconda-18.23-1.fc18: * should fix your issue, * was pushed to the Fedora 18 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing anaconda-18.23-1.fc18' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2012-17509/anaconda-18.23-1.fc18 then log in and leave karma (feedback). anaconda-18.24-1.fc18 has been submitted as an update for Fedora 18. https://admin.fedoraproject.org/updates/anaconda-18.24-1.fc18 Package anaconda-18.24-1.fc18: * should fix your issue, * was pushed to the Fedora 18 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing anaconda-18.24-1.fc18' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2012-17543/anaconda-18.24-1.fc18 then log in and leave karma (feedback). anaconda-18.25-1.fc18 has been submitted as an update for Fedora 18. https://admin.fedoraproject.org/updates/anaconda-18.25-1.fc18 Can someone please re-test and confirm the fix with Beta TC7? Thanks! Do you have an Intel multipath environment to test on, Adam? The ppc builds are currently broken. No. tflink may have access to one. anaconda-18.26-1.fc18 has been submitted as an update for Fedora 18. https://admin.fedoraproject.org/updates/anaconda-18.26-1.fc18 I did a default install on an i686 machine with a clean promise RAID1 (dmraid). Autopart selection went off without a hitch and the install got past the partitioning and packaging phases. However, I hit #873463 before the install finished. Since this bug manifested before this point, I'm moving to VERIFIED. 18.26 went stable. Closing. (Bodhi closing of bugs when updates go stable is currently broken). |
Created attachment 629046 [details] Reclaim Disk Space dialog Description of problem: When installing Fedora 18, I see no free space to reclaim on a multipath system with anaconda-18.17-1.fc18.