Description of problem: Cannot install F22, since installer does not recognise my RAID devices from F21 installation. Burned to DVD and verified image Fedora-Live-Workstation-x86_64-22-3.iso and booted from disk. After a very long boot time I did try to upgrade my F21 system. Installer shows for some seconds a "Sorry, ... problem ..." message but than shows normal install screens. The Installer sees all single partitions except my RAID device. From the working F21 system this device looks as follows: # mdadm -E /dev/sd[b-c] /dev/sdb: MBR Magic : aa55 Partition[0] : 4294967295 sectors at 1 (type ee) /dev/sdc: MBR Magic : aa55 Partition[0] : 4294967295 sectors at 1 (type ee) # mdadm -E /dev/sd[b-c]1 /dev/sdb1: Magic : a92b4efc Version : 1.2 Feature Map : 0x0 Array UUID : b145e209:0ec81538:744622c4:c33ae860 Name : bl4ckh0l4.s4y.no:opt_yachad00 Creation Time : Fri Jan 16 16:05:19 2015 Raid Level : raid0 Raid Devices : 2 Avail Dev Size : 5000290304 (2384.32 GiB 2560.15 GB) Data Offset : 262144 sectors Super Offset : 8 sectors Unused Space : before=262056 sectors, after=0 sectors State : clean Device UUID : 7f1b3ea8:259fbe15:c7e547c6:9408f126 Update Time : Fri Jan 16 16:05:19 2015 Bad Block Log : 512 entries available at offset 72 sectors Checksum : a69b8d5 - correct Events : 0 Chunk Size : 512K Device Role : Active device 0 Array State : AA ('A' == active, '.' == missing, 'R' == replacing) /dev/sdc1: Magic : a92b4efc Version : 1.2 Feature Map : 0x0 Array UUID : b145e209:0ec81538:744622c4:c33ae860 Name : bl4ckh0l4.s4y.no:opt_yachad00 Creation Time : Fri Jan 16 16:05:19 2015 Raid Level : raid0 Raid Devices : 2 Avail Dev Size : 5000290304 (2384.32 GiB 2560.15 GB) Data Offset : 262144 sectors Super Offset : 8 sectors Unused Space : before=262056 sectors, after=0 sectors State : clean Device UUID : 6bd4a9ec:204bf024:33c224ad:45393e39 Update Time : Fri Jan 16 16:05:19 2015 Bad Block Log : 512 entries available at offset 72 sectors Checksum : 57312ada - correct Events : 0 Chunk Size : 512K Device Role : Active device 1 Array State : AA ('A' == active, '.' == missing, 'R' == replacing)
The installation process using the server edition with Fedora-Server-DVD-x86_64-22.iso works fine. RAID partitions will correctly be recognised.
If the Server is working and Live isn't then something must be missing from the kickstart used to create the Live images.
Odd. I can't think of what this might be. Can you tell us more about the raid? Model/make/etc... Could you attach a 'dmesg' from both f22 live and f22 install media boots?
Now, I did the following: o used Server-DVD to install a system o installed with Live-Workstation on a notebook without problems (no raid) o installed on the server all the packages from WS installation from notebook o changed boot target and some stuff (first of all 'dnf remove xorg-x11-drv-libinput-0.9.0-1.fc22.x86_64' since my mouse is nearly not usable under F22) and have now a shiny new F22 system! :-) Repeated the live install and I've got the same error messages and no raid detected. Saved some stuff from this live session incl. the abrt report from /var/tmp/abrt/ and will attach it now.
Created attachment 1031757 [details] var_live_raid_error_ws.tar.bz2
Created attachment 1031758 [details] proc_live_raid_error_ws.tar.bz2
Created attachment 1031759 [details] partitions /dev/sda-c
Created attachment 1031760 [details] etc_live_raid_error_ws.tar.bz2
Created attachment 1031761 [details] mdadm_live_raid_error_ws.tar.bz2
Created attachment 1031762 [details] rpm --query --all --qf "%-30{NAME} - %{VERSION}\n" | sort
Created attachment 1031763 [details] rpm -qa --queryformat='%{NAME}\n' | sort
ok, this is intel fw raid. I wonder if it's: https://bugzilla.redhat.com/show_bug.cgi?id=1219264
But, interestingly, why does the server edition not have the same bug? I've had this (software raid) configuration since F20 at least. And it always worked without problems.
No idea, but you can see AdamW saw the same thing in that bug... server/install dvd worked, live didn't.
With the final F22 Workstation installation image (x86, 64 bit) I've the same problem for my existing F21 installation based on regular software RAID-1 (md*). When I boot the Workstation image (USB stick) and select "Install to Harddisk", first some mysterious crash occurs (Oops...) but eventually the Installer starts up. Unfortunately, when selecting the destination devices, no software RAID devices (md*) from my existing F21 installation will show up. So there's nothing I can install F22 on. There's no hardware-based RAID involved or any motherboard RAID functions. Works fine with the Server Image. Looks similar to bug #1225671
Just tried another install on a Thinkpad X300 with Fedora-Live-Workstation. This time and on that specific hardware it stops with an error message "Oh No Something went wrong. The system encountered a problem and can't recover.". Applying the same procedure as layed out in "comment 4" I could finally install F22 workstation. So far the workstation install worked for me on a single hardware (Thinkpad T440s) out of the box, On all outher notebooks and pc's it just failed. ;-\ On that Thinkpad X300 there is no information in /var/tmp/abrt. I'll attach the /var/lob/boot.log and dmesg.
Created attachment 1034360 [details] thinkpad x300 /var/log/boot.log
Created attachment 1034361 [details] thinkpad x300 dmesg
Essentially, this problem has not been fixed for Fedora 23. Although RAID devices no longer cause crashes in Anaconda and are somehow recognized, they are flagged es "Unknown" (unknown size, unknown label, unknown formatting etc) and cannot be used during the Installation process. Several bug reports where filed for this issue. All ignored. Does that mean, Software RAID is no longer supported officially and users have to switch distribution?
Yep, have seen it in F23. Workstation image crashed, and i've used the server edition to update systems. It is not just RAID. Encrypted partitions are not liked as well. :-\
I can not boot F22 kernels on mdadm raid device with LVM over it. I am dropped to dracut shell. blkid shows correctly all the devices, but they are not assembled, the of course the LVM which is over the raid fails too + command -v lvm + lvm pvdisplay + lvm vgdisplay + lvm lvdisplay + command -v dmsetup + dmsetup ls --tree No devices found + cat /proc/mdstat Personalities : unused devices: <none> As I can reproduce this on every boot with F22 kernel (F21 kernel works!) I can test whatever will make sense. I'd like to point out, that grub.cfg generated by grub2-mkconfig fails badly too, but this may not be related.
Just been bitten by this bug for Fedora 23. It is ridiculous that this has not been fixed since Fedora 22 it is pretty basic required functionality and shows the very limited testing that Anaconda/install has had.
Cool, thanks for that!
Sorry, spent almost a day trying to work around this and was quite frustrated. I also tried Fedora-Server-DVD-x86_64-23.iso and that is ok (but not very suitable for our use). The contents of this is significantly different to the workstation live images, not sure why this should be so ? Have had a play with trying to debug what is happening, but haven't the knowledge or time to delve deeply. I have run the "liveinst" script from a GUI terminal and had a look at the files in /tmp. Some things: 1. The raid devices are not assembled/started when anaconda is running. In fact the raid arrays are stopped when "liveinst" is run (If I start up the arrays manually before). 2. I see messages like: DEBUG blivet: MDRaidArrayDevice.readCurrentSize: path: /dev/md/4 ; exists: True ; sysfsPath: ; DEBUG blivet: existing RAID raid1 size == 0 B in /tmp/storage.log There is no /dev/md/4 as the raid devices is shutdown. 3. If I run "anaconda" directly I get what looks like a "future" anaconda with lots of warning messages about being development software etc. However if I proceed to the manual disk partitioning page it does see the raid partitions with the correct size. I haven't done further with this. I will attach the /tmp/*.log files in case they are useful.
Created attachment 1103567 [details] Anaconda log files
For my case from Comment 21 I have "resolution" here: https://bugzilla.redhat.com/show_bug.cgi?id=1286464
The Fedora Server ISO image can be used as a workaround. Either perform a "minimal install" (Server) and once the system is installed use "dnf" to switch to the Workstation distribution. Or select the Workstation installation within the Server installer. However, both ways require a good Internet connection as they both download all packages via network. (Unfortunately, the Server ISO doesn't allow to use a local Workstation ISO image file as source.) Furthermore, the packages are a little bit different to a real Workstation installation. Some additional work is required after installation to finally have a proper Workstation environment. Anyway, this bug should be fixed. It's strange that Fedora fails so badly on RAID devices or encrypted volumes. It worked well up to F21. Question is if there are any intentions to fix it, and if not, will upcoming Red Hat releases (which are based on Fedora) also stop support for RAID and encrypted devices?
Updating based on comments that this still occurs on F23.
Based on the comment that this works with the Server installer, I expect that using the Workstation netinst ISO should serve as another workaround. Download from https://getfedora.org/en/workstation/download/ -- look in the right column -- or direct link https://download.fedoraproject.org/pub/fedora/linux/releases/23/Workstation/x86_64/iso/Fedora-Workstation-netinst-x86_64-23.iso Can someone confirm?
Matthew, the Workstation netinst ISO correctly detects the software RAID and offers the mapped block devices as targets for installation.
By utilizing the F23 netinstall image, I was able to perform an LVM on Software RAID install.
So 8 months on and still no fix for this ?
I encountered this problem trying to install F23 on a fake raid system (AMD SB950 with mirrored disks). I tried installing the F23 scientific spin, the F23 workstation spin and the F23 net install spin. None of these would recognize the raid drive. No drives except the live USB were visible for installation. I tried using wipefs to clear the hard drives and recreate the raid logical volume, but that was a no go as well. FWIW: the newly released Kubuntu 16.04 live install iso does recognize the raid and installs OK so I know the raid volume and the disks are not the problem. In the end I had to delete the raid logical volume and install F23 on a single drive, which worked fine. I hope this problem can get some attention, as I would like to stick with Fedora and eventually migrate back to a mirrored raid system. Thanks for any help you can give me.
Erik J: Comments 28 through 31 provide a workaround. The normal workstation ISO does not work for softraid (mdadm) disk layouts, but the netinstall ISO does (I tried the workstation one and had success). Use this method if you want to stick with Fedora. Terry Barnaby: I wouldn't expect a fix until the next release (F24). Since F24 final release is expected in June 2016 I would question a fix will make that release. https://fedoraproject.org/wiki/Releases/24/Schedule I don't see any mention of "raid" in the common bugs page (not that its absence necessarily means anything). https://fedoraproject.org/wiki/Common_F24_bugs
silvertip257: Thanks for the reply. Please note that I did try the netinstall ISO and it *did not* recognize the raid, which is why I posted the comment. I read the entire thread carefully and tried the workarounds mentioned. When the netinstall did not work I wanted to bring it to people's attention. I am happy to try other options or perform specific tests if it will help. Thanks.
This bug still exists in F24. Neither Workstation ISO, nor Workstation netinst recognize softraid. I have read the comments and work-arounds in the thread, but they do not work for my HW.
A clarification, which I think explains why the work-around described above does not work in my case: I am using firmware raid, which is not the same as mdadm. The problem is that anaconda does not detect the firmware raid. I am sure the work-around is fine for mdadm, but it does not work for firmware raid. Sorry for any confusion I may have caused.
I am also noticing this problem in F24. I have an Intel Rapid Storage **HARDWARE** raid that is mirroring my two devices. In Fedora 21 the RAID was properly detected and assembled and I was able to create a partition on the RAID-ed device for installation of Fedora. In Fedora 24 workstation live the device is not properly detected at all. It's really quite a mess. The following commands were all issued on Fedora 24 in a live session: [root@localhost ~]# cat /proc/mdstat Personalities : md127 : inactive sda[0](S) 3028 blocks super external:imsm unused devices: <none> [root@localhost ~]# mdadm --detail-platform Platform : Intel(R) Rapid Storage Technology Version : 13.0.0.2075 RAID Levels : raid0 raid1 raid10 raid5 Chunk Sizes : 4k 8k 16k 32k 64k 128k 2TB volumes : supported 2TB disks : supported Max Disks : 6 Max Volumes : 2 per array, 4 per controller I/O Controller : /sys/devices/pci0000:00/0000:00:1f.2 (SATA) Port0 : /dev/sda (WD-WCC4NKTRPZZ0) Port2 : - non-disk device (HP DVD Writer 1160d) - Port1 : /dev/sdb (WD-WCC4NRNX77SK) Port3 : - no device attached - [root@localhost ~]# man mdadm [root@localhost ~]# mdadm -E /dev/sda1 mdadm: cannot open /dev/sda1: No such file or directory [root@localhost ~]# mdadm -E /dev/sda /dev/sda: Magic : Intel Raid ISM Cfg Sig. Version : 1.1.00 Orig Family : ab616185 Family : ab616185 Generation : 004f8499 Attributes : All supported UUID : e97d24bf:d4c05d9f:899652c4:01270fbd Checksum : b3cddd5e correct MPB Sectors : 1 Disks : 2 RAID Devices : 1 Disk00 Serial : WD-WCC4NKTRPZZ0 State : active Id : 00000000 Usable Size : 3907023112 (1863.01 GiB 2000.40 GB) [Main]: UUID : dac43f62:7f66c7ad:5a0238af:fcd2ff4f RAID Level : 1 Members : 2 Slots : [UU] Failed disk : none This Slot : 0 Array Size : 3907022848 (1863.01 GiB 2000.40 GB) Per Dev Size : 3907023112 (1863.01 GiB 2000.40 GB) Sector Offset : 0 Num Stripes : 15261808 Chunk Size : 64 KiB Reserved : 0 Migrate State : idle Map State : normal Dirty State : clean Disk01 Serial : WD-WCC4NRNX77SK State : active Id : 00000001 Usable Size : 3907023112 (1863.01 GiB 2000.40 GB) [root@localhost ~]# [root@localhost ~]# [root@localhost ~]# mdadm -E /dev/sdb /dev/sdb: Magic : Intel Raid ISM Cfg Sig. Version : 1.1.00 Orig Family : ab616185 Family : ab616185 Generation : 004f8499 Attributes : All supported UUID : e97d24bf:d4c05d9f:899652c4:01270fbd Checksum : b3cddd5e correct MPB Sectors : 1 Disks : 2 RAID Devices : 1 Disk01 Serial : WD-WCC4NRNX77SK State : active Id : 00000001 Usable Size : 3907023112 (1863.01 GiB 2000.40 GB) [Main]: UUID : dac43f62:7f66c7ad:5a0238af:fcd2ff4f RAID Level : 1 Members : 2 Slots : [UU] Failed disk : none This Slot : 1 Array Size : 3907022848 (1863.01 GiB 2000.40 GB) Per Dev Size : 3907023112 (1863.01 GiB 2000.40 GB) Sector Offset : 0 Num Stripes : 15261808 Chunk Size : 64 KiB Reserved : 0 Migrate State : idle Map State : normal Dirty State : clean Disk00 Serial : WD-WCC4NKTRPZZ0 State : active Id : 00000000 Usable Size : 3907023112 (1863.01 GiB 2000.40 GB) [root@localhost ~]# [root@localhost ~]# [root@localhost ~]# mdadm -E /dev/sdb1 /dev/sdb1: MBR Magic : aa55 Partition[0] : 1836016416 sectors at 1936269394 (type 4f) Partition[1] : 544437093 sectors at 1917848077 (type 73) Partition[2] : 544175136 sectors at 1818575915 (type 2b) Partition[3] : 54974 sectors at 2844524554 (type 61) I will submit attachments for various other things. Supposedly there is a workaround but I could not understand it from this thread. Also note that: https://bugzilla.redhat.com/show_bug.cgi?id=1219264 https://bugzilla.redhat.com/show_bug.cgi?id=1172426 both appear to be related.
Created attachment 1172604 [details] Fedora 21 mdstat and mdadm.conf I did not create the mdadm.conf. I am assuming the installer created it when I installed F21.
Created attachment 1172605 [details] Fedora 21 fdisk -l output
Created attachment 1172606 [details] Fedora 21 dmesg output
Created attachment 1172607 [details] Fedora 24 dmesg output
Created attachment 1172608 [details] Fedora 24 fdisk -l output
Note that while the RAID device is not detected, one of the member disks is detected and its partitions are shown: [root@localhost ~]# fdisk -l /dev/sdb Disk /dev/sdb: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disklabel type: dos Disk identifier: 0x0001f847 Device Boot Start End Sectors Size Id Type /dev/sdb1 * 2048 206847 204800 100M 7 HPFS/NTFS/exFAT /dev/sdb2 206848 2977521663 2977314816 1.4T 7 HPFS/NTFS/exFAT /dev/sdb3 2977521664 3790045183 812523520 387.5G 8e Linux LVM Additionally, the PV/LV/VG information is available, too: [root@localhost ~]# pvs PV VG Fmt Attr PSize PFree /dev/sdb3 fedora lvm2 a-- 387.44g 4.00m [root@localhost ~]# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert home fedora -wi-a----- 279.40g root fedora -wi-a----- 93.13g swap fedora -wi-ao---- 14.90g [root@localhost ~]# vgs VG #PV #LV #SN Attr VSize VFree fedora 1 3 0 wz--n- 387.44g 4.00m [root@localhost ~]# vgdisplay --- Volume group --- VG Name fedora System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 4 VG Access read/write VG Status resizable MAX LV 0 Cur LV 3 Open LV 1 Max PV 0 Cur PV 1 Act PV 1 VG Size 387.44 GiB PE Size 4.00 MiB Total PE 99184 Alloc PE / Size 99183 / 387.43 GiB Free PE / Size 1 / 4.00 MiB VG UUID wb4KFn-nxCJ-beqy-nXjA-u2N0-Gnv7-dfRCNB [root@localhost ~]# pvdisplay --- Physical volume --- PV Name /dev/sdb3 VG Name fedora PV Size 387.44 GiB / not usable 4.00 MiB Allocatable yes PE Size 4.00 MiB Total PE 99184 Free PE 1 Allocated PE 99183 PV UUID J1diTf-CZLP-D8dW-TGuT-LOzR-y3s8-8oQs0I [root@localhost ~]# lvdisplay --- Logical volume --- LV Path /dev/fedora/root LV Name root VG Name fedora LV UUID BeGdLK-r9iu-kZgf-RdpM-vRFM-iQS3-E2OhK2 LV Write Access read/write LV Creation host, time localhost, 2015-01-11 21:29:11 -0500 LV Status available # open 0 LV Size 93.13 GiB Current LE 23842 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 253:0 --- Logical volume --- LV Path /dev/fedora/home LV Name home VG Name fedora LV UUID 19Epia-1Rj6-ulxP-Lu25-0UOS-tBDu-woicRw LV Write Access read/write LV Creation host, time localhost, 2015-01-11 21:29:16 -0500 LV Status available # open 0 LV Size 279.40 GiB Current LE 71526 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 253:1 --- Logical volume --- LV Path /dev/fedora/swap LV Name swap VG Name fedora LV UUID P7s3Y6-sQkW-Gwz3-4Qas-0Fjx-7gOZ-HRFnLL LV Write Access read/write LV Creation host, time localhost, 2015-01-11 21:29:28 -0500 LV Status available # open 2 LV Size 14.90 GiB Current LE 3815 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 253:2 But **none** of these devices are shown as being available for the installer to use.
So, I think the issue actually lies outside of Anaconda in my particular case. By adding "rd.auto" to the startup of the live image, the RAID array was properly assembled and then available for use as an installation target. F24 then installed with no issue.
Hello again. I have tried the rd.auto method that Erik Jacobs mentions above, as well as rd.dm=1, but am having no luck. I found 2 other bug reports that are similar, but without any resolution or workarounds that work for me: https://bugzilla.redhat.com/show_bug.cgi?id=1219264 https://bugzilla.redhat.com/show_bug.cgi?id=1225618 I have tried just about everything imaginable, including wiping the disks with nwipe and creating a fresh raid set. I have tried installing Fedora Workstation, Server, and tried netinst, but still no luck. I have uploaded various outputs from my install attempts below. Blivet complains that my disks do not appear to be part of an array, but later finds the raid device. the raid device is detected using dmraid as well. I am curious if the blivet warning is because the devices show up as "mirror" for RAID-1, which is not quite the same as part of an array as in RAID-5 or 10. If I install gnome-disk-utility onto the live version it correctly detects the RAID. Please point me in the right direction if this is not the right component under which this should be reported or if I should submit a new bug report. I am happy to help with any other tests you would like me to run to debug this. My HW is using the AMD SB 950 chipset, which evidently is using a Promise FastTrack firmware RAID controller which is enabled in the BIOS. The RAID functionality works, as I have previously installed Linux Mint and Kubuntu on it, so it is likely not a problem with the RAID set. The install has failed for both F23 and F24 (works fine for individual disk installs, though). I have not tried any previous versions, although the 1219264 bug report seems to indicate that RAID sets were correctly detected previously for F20. Thanks for any help or guidance you can give me. Erik J
Created attachment 1179314 [details] F24 Anaconda log
Created attachment 1179315 [details] F24 storage log
Created attachment 1179317 [details] F24 fdisk output
Created attachment 1179318 [details] F24 dmraid output
Created attachment 1179319 [details] F24 smartctl output
vpodzime, do you know why libblockdev doesn't report any arrays for the promise fasttrack members in comment 46? It seems like dmraid sees them, but libblockdev's call to libdmraid does not.
Hi David and vpodzime, I had a chance to debug this last weekend. Unfortunately, my notes are at home, so I will have to update this post later tonight with more details. I tracked everything down to a particular routine in blivet where it tries to add the array but fails because it does not return a valid device from getDeviceByName. Here is the pertinent output in storage.log where the problem occurs (see below). I am happy to help you troubleshoot this if you can give me a little guidance (my expertise is in signal processing and scientific sw). I already tried hacking a bit in blivet to try to force it to load the array into the device tree, but got stuck later on when it wanted the format type and some other info that was missing which caused blivet to bail. Thanks, Erik J. 09:04:39,320 INFO blivet: scanning pdc_iffcgcja (/sys/devices/virtual/block/dm-0)... 09:04:39,320 DEBUG blivet: DeviceTree.getDeviceByName: name: pdc_iffcgcja ; hidden: False ; incomplete: False ; 09:04:39,321 DEBUG blivet: DeviceTree.getDeviceByName returned None 09:04:39,321 INFO blivet: pdc_iffcgcja is a device-mapper device 09:04:39,321 DEBUG blivet: Populator.addUdevDMDevice: name: pdc_iffcgcja ; 09:04:39,326 DEBUG blivet: DeviceTree.getDeviceByName: name: sda ; hidden: False ; incomplete: False ; 09:04:39,327 DEBUG blivet: DeviceTree.getDeviceByName returned existing 465.76 GiB disk sda (0) with existing dmraidmember 09:04:39,328 DEBUG blivet: DeviceTree.getDeviceByName: name: sdb ; hidden: False ; incomplete: False ; 09:04:39,329 DEBUG blivet: DeviceTree.getDeviceByName returned existing 465.76 GiB disk sdb (6) with existing dmraidmember 09:04:39,329 DEBUG blivet: DeviceTree.getDeviceByName: name: pdc_iffcgcja ; hidden: False ; incomplete: False ; 09:04:39,330 DEBUG blivet: DeviceTree.getDeviceByName returned None 09:04:39,330 DEBUG blivet: lvm filter: adding pdc_iffcgcja to the reject list 09:04:39,330 WARN blivet: ignoring dm device pdc_iffcgcja 09:04:39,330 DEBUG blivet: no device obtained for pdc_iffcgcja
A brief update: In blivet:populator.py the method addUdevDevice is called to add the device to the tree (this is where the log note "scanning pdc_iffcgcja" is generated). In the method it calls self.getDeviceByName to get the device, which returns "None". Further down in the method it goes through a long if-elif statement to either lookup or create the device: # # The first step is to either look up or create the device # device_added = True if device: device_added = False elif udev.device_is_loop(info): log.info("%s is a loop device", name) device = self.addUdevLoopDevice(info) elif udev.device_is_dm_mpath(info) and \ not udev.device_is_dm_partition(info): log.info("%s is a multipath device", name) device = self.addUdevMultiPathDevice(info) elif udev.device_is_dm_lvm(info): log.info("%s is an lvm logical volume", name) device = self.addUdevLVDevice(info) elif udev.device_is_dm(info): log.info("%s is a device-mapper device", name) device = self.addUdevDMDevice(info) elif udev.device_is_md(info) and not udev.device_get_md_container(info): log.info("%s is an md device", name) device = self.addUdevMDDevice(info) elif udev.device_is_cdrom(info): log.info("%s is a cdrom", name) device = self.addUdevOpticalDevice(info) elif udev.device_is_disk(info): device = self.addUdevDiskDevice(info) elif udev.device_is_partition(info): log.info("%s is a partition", name) device = self.addUdevPartitionDevice(info) else: log.error("Unknown block device type for: %s", name) return if not device: log.debug("no device obtained for %s", name) return The device-mapper branch is taken and the method addUdevDMDevice is called. This returns "none" for the device so the original addUdevDevice method returns no device. The addUdevDMDevice method calls self.getDeviceByName to get the device which, as before, returns "None" (it simply compares the name "pdc_iffcgcja" to all the devices already in the _devices[] array and no match is found). Later in the method we reach this code snippet: # if we get here, we found all of the slave devices and # something must be wrong -- if all of the slaves are in # the tree, this device should be as well if device is None: lvm.lvm_cc_addFilterRejectRegexp(name) log.warning("ignoring dm device %s", name) return device The device is still "None" so no device is ever created or loaded and the method returns "None", so no further info on this device is made available to anaconda. I don't know enough about how all the udev device things work in blivet, but my guess is that addUdevDMDevice should create the device and add it to the tree with the appropriate meta-data. As I mentioned before, I am happy to help trouble shoot this if you can give me guidance on what to test. Thanks, Erik J.
(In reply to David Lehman from comment #52) > vpodzime, do you know why libblockdev doesn't report any arrays for the > promise fasttrack members in comment 46? It seems like dmraid sees them, but > libblockdev's call to libdmraid does not. I have no idea. I'd need the particular piece of HW and do some step-by-step debugging to see what's going on.
Erik, could you please try to compare the results of the following? 1) install python-pyblock 2) run 'sudo python' and: >>> import block >>> block.getRaidSetFromRelatedMem(name=NAME_OF_ONE_RAID_MEMBER) 3) run 'sudo python3' >>> from gi.repository import BlockDev; BlockDev.init() >>> BlockDev.dm.get_member_raid_sets(name=NAME_OF_ONE_RAID_MEMBER) libblockdev's get_member_raid_sets() function was written based on the pyblock's codebase so I'd like to see if they give the same results. Thanks!
Hi Vratislav, Here is the output you requested. First a dmraid output so you can see the devices and names: [root@shadowfax-amd-single erikj]# dmraid -r /dev/sdb: pdc, "pdc_bbcbddjca", mirror, ok, 976562432 sectors, data@ 0 /dev/sda: pdc, "pdc_bbcbddjca", mirror, ok, 976562432 sectors, data@ 0 Output from python 2.7.12 and block: [erikj@shadowfax-amd-single ~]$ sudo python Python 2.7.12 (default, Jul 18 2016, 10:55:51) [GCC 6.1.1 20160621 (Red Hat 6.1.1-3)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import block >>> block.getRaidSetFromRelatedMem(name="/dev/sda") [] >>> block.getRaidSetFromRelatedMem(name="/dev/sdb") [] >>> block.getRaidSetFromRelatedMem(name="pdc_bbcbddjca") [] >>> Output from python 3 and BlockDev: [erikj@shadowfax-amd-single ~]$ sudo python3 Python 3.5.1 (default, Jul 10 2016, 20:36:01) [GCC 6.1.1 20160621 (Red Hat 6.1.1-3)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> from gi.repository import BlockDev; BlockDev.init() __main__:1: PyGIWarning: BlockDev was imported without specifying a version first. Use gi.require_version('BlockDev', '1.0') before import to ensure that the right version gets loaded. True >>> BlockDev.dm.get_member_raid_sets(name="/dev/sda") [] >>> BlockDev.dm.get_member_raid_sets(name="/dev/sdb") [] >>> BlockDev.dm.get_member_raid_sets(name="pdc_bbcbddjca") [] >>> In both cases they return nothing. I am not sure if I am using the correct format for the device names, please confirm. Thanks, Erik
Could you please also try just "sda" and "sdb"?
Sorry, don't know why I didn't think of that. OK, now we have output: Python 2/7 and block: >>> block.getRaidSetFromRelatedMem(name="sda") [<block.device.RaidSet instance at 0x7faad89f0560>] >>> block.getRaidSetFromRelatedMem(name="sdb") [<block.device.RaidSet instance at 0x7faad89f0638>] Python 3 and BlockDev: >>> BlockDev.dm.get_member_raid_sets(name="sda") ['pdc_bbcbddjca'] >>> BlockDev.dm.get_member_raid_sets(name="sdb") ['pdc_bbcbddjca']
Maybe related: https://bugzilla.redhat.com/show_bug.cgi?id=1361611 At least your debugging put me on track on where to look, so kudos for that.
(In reply to Erik J from comment #59) > Sorry, don't know why I didn't think of that. OK, now we have output: > > Python 2/7 and block: > > >>> block.getRaidSetFromRelatedMem(name="sda") > [<block.device.RaidSet instance at 0x7faad89f0560>] > >>> block.getRaidSetFromRelatedMem(name="sdb") > [<block.device.RaidSet instance at 0x7faad89f0638>] > > Python 3 and BlockDev: > > >>> BlockDev.dm.get_member_raid_sets(name="sda") > ['pdc_bbcbddjca'] > >>> BlockDev.dm.get_member_raid_sets(name="sdb") > ['pdc_bbcbddjca'] Thanks! Looks like it could possibly be this stupid mistake: https://github.com/rhinstaller/blivet/pull/473
Hi Vratislav, I made this change to populator.py per your github link: # diff populator.py populator.py~ 1256c1256 < rs_names = blockdev.dm.get_member_raid_sets(name, uuid, major, minor) --- > rs_names = blockdev.dm.get_member_raid_sets(uuid, name, major, minor) Then I ran anaconda, which aborted with the following error: anaconda 24.13.7-1 exception report Traceback (most recent call first): File "/usr/lib64/python3.5/site-packages/gi/overrides/BlockDev.py", line 429, in wrapped raise transform[1](msg) File "/usr/lib/python3.5/site-packages/blivet/populator.py", line 1269, in handleUdevDMRaidMemberFormat blockdev.dm.activate_raid_set(rs_name) File "/usr/lib/python3.5/site-packages/blivet/populator.py", line 1487, in handleUdevDeviceFormat self.handleUdevDMRaidMemberFormat(info, device) File "/usr/lib/python3.5/site-packages/blivet/populator.py", line 773, in addUdevDevice self.handleUdevDeviceFormat(info, device) File "/usr/lib/python3.5/site-packages/blivet/populator.py", line 1701, in _populate self.addUdevDevice(dev) File "/usr/lib/python3.5/site-packages/blivet/populator.py", line 1638, in populate self._populate() File "/usr/lib/python3.5/site-packages/blivet/devicetree.py", line 555, in populate self._populator.populate(cleanupOnly=cleanupOnly) File "/usr/lib/python3.5/site-packages/blivet/blivet.py", line 279, in reset self.devicetree.populate(cleanupOnly=cleanupOnly) File "/usr/lib/python3.5/site-packages/blivet/osinstall.py", line 1157, in storageInitialize storage.reset() File "/usr/lib64/python3.5/threading.py", line 862, in run self._target(*self._args, **self._kwargs) File "/usr/lib64/python3.5/site-packages/pyanaconda/threads.py", line 253, in run threading.Thread.run(self, *args, **kwargs) gi.overrides.BlockDev.DMError: Failed to activate the RAID set 'pdc_bbcbddjca' I have not had time to try to troubleshoot this yet. I am attaching the anaconda crash output, anaconda.log and storage.log below. Thanks, Erik J
Created attachment 1186853 [details] anaconda exception report
Created attachment 1186854 [details] anaconda.log
Created attachment 1186855 [details] storage.log
This is still an issue with F25 Alpha Workstation Live. Reproducer: * Install F24, standard partitions, onto two disks (raid 0). I used Workstation Live for that. * Boot the system to verify it works. * Boot F25 Workstation Live, go to custom partition, see that partitions are listed as unknown and cannot be used. Setting a better component (blivet, since the patch went into blivet). Proposing as a Beta blocker: "When using the custom partitioning flow, the installer must be able to: Correctly interpret, and modify as described below, any disk with a valid ms-dos or gpt disk label and partition table containing ext4 partitions, LVM and/or btrfs volumes, and/or software RAID arrays at RAID levels 0, 1 and 5 containing ext4 partitions "
As Kevin mentioned last year, this and https://bugzilla.redhat.com/show_bug.cgi?id=1219264 may well be the same , and I believe that for me involved one of the disks the RAID set was built out of having stale LVM metadata on it. I'd suggest checking the installer logs to see if it seems to be finding LVM volumes unexpectedly, and maybe running 'lvdisplay' and 'vgdisplay' and friends...
(In reply to Kamil Páral from comment #66) > This is still an issue with F25 Alpha Workstation Live. Reproducer: Can you attach the logs?
Created attachment 1195453 [details] storage.log 20160829 Just like in kparal's comment 66. What I did was: 1. create two new qcow2 files 2. boot f24 installer, and do a single root mount point, XFS on mdadm raid0 installation, nothing else. It boots after installation fine. 3. boot f25 installer, and I get an "unknown" item in manual partitioning, under which is a 0byte "root" unknown item.
Created attachment 1195454 [details] program.log 20160829 program.log that goes with storage.log above
Created attachment 1195458 [details] anaconda.log 20160829 anaconda.log that goes with comment 69 and 70
Created attachment 1195460 [details] screenshot 20160829 screenshot goes with comment 69, 70, 71
It looks like there are two things going on here: 1. the live image is not auto-activating md as all other system variants do 2. blivet has lost the ability to activate them since they're auto-activated everywhere except, apparently, on fedora live images Someone needs to decide if fedora live images are not activating md by design or in error. It's probably a bug that blivet doesn't auto-activate them as needed, but I'm not particularly moved by arbitrary variations in the various installation environments.
IIRC it's intentional that we don't automatically bring up any local storage when booting live, because we want to avoid any possibility of unexpected changes to it. But I'm not 100% sure. It would probably be a good idea to boot some old lives and see if they were auto-activating RAID sets or not.
(In reply to Adam Williamson from comment #74) > IIRC it's intentional that we don't automatically bring up any local storage > when booting live, because we want to avoid any possibility of unexpected > changes to it. But I'm not 100% sure. It would probably be a good idea to > boot some old lives and see if they were auto-activating RAID sets or not. If anything, I think having the storage active by default is likely to _prevent_ unintentional changes to it (think EBUSY). Either way, it would be good to communicate such a change to the installer team.
I wasn't actually saying it was a change... thinking a bit more, though, it may have changed a few years back when dracut changed its behaviour regarding auto-activating RAID sets...
It's active before I launch the installer: [liveuser@localhost ~]$ su [root@localhost liveuser]# cat /proc/mdstat Personalities : [raid0] md127 : active raid0 vdb1[1] vda1[0] 18890752 blocks super 1.2 512k chunks unused devices: <none> From the program.log: 14:16:14,827 INFO program: Running [14] mdadm --stop /dev/md/root ...
Thanks for pointing that out -- I overlooked 'root' in the initial device list. You ran anaconda at least twice in that live session. What happened the first time?
Hmm, or is all that previous output from anaconda-cleanup?
blivet does indeed deactivate /dev/md/root as part of anaconda-cleanup. This is not necessarily desirable _before_ the installation, but that's another matter. The reason that shouldn't matter is anaconda then does 'udevadm trigger --subsystem=block --action=change', which should activate the md array again. The reason this doesn't work out is probably related to a relic from the past: udevadm control --env=ANACONDA=1 Chris, can you please try the f25 installation again but, this time, comment out the following lines: /usr/lib/udev/rules.d/64-md-raid-assembly.rules:ENV{ANACONDA}=="?*", GOTO="md_inc_end" /usr/lib/udev/rules.d/65-md-incremental.rules:ENV{ANACONDA}=="?*", GOTO="md_end" (Just put a '#' at the start of each line.) Then try the installation and see if things improve. Thanks.
(In reply to David Lehman from comment #78) > Thanks for pointing that out -- I overlooked 'root' in the initial device > list. You ran anaconda at least twice in that live session. What happened > the first time? There was only one launch of anaconda in that live session. Is there any chance the udisks2>storaged change is involved in any confusion? (In reply to David Lehman from comment #80) > Chris, can you please try the f25 installation again but, this time, comment > out the following lines: > > /usr/lib/udev/rules.d/64-md-raid-assembly.rules:ENV{ANACONDA}=="?*", > GOTO="md_inc_end" > /usr/lib/udev/rules.d/65-md-incremental.rules:ENV{ANACONDA}=="?*", > GOTO="md_end" So boot the live media, then comment out these lines, then launch the installer?
(In reply to Chris Murphy from comment #81) > There was only one launch of anaconda in that live session. Is there any > chance the udisks2>storaged change is involved in any confusion? No, I just forgot all about anaconda-cleanup. > So boot the live media, then comment out these lines, then launch the > installer? Correct.
(In reply to David Lehman from comment #80) > /usr/lib/udev/rules.d/64-md-raid-assembly.rules:ENV{ANACONDA}=="?*", > GOTO="md_inc_end" > /usr/lib/udev/rules.d/65-md-incremental.rules:ENV{ANACONDA}=="?*", > GOTO="md_end" No change.
Created attachment 1196013 [details] program.log 20160830 program.log from comment 83
Created attachment 1196014 [details] storage.log 20160830 storage.log from comment 83
If you are still willing, please try once more, but this time run the following command after changing the rules and before starting anaconda: udevadm control --reload-rules Thanks again.
(In reply to David Lehman from comment #86) > If you are still willing, please try once more, but this time run the > following command after changing the rules and before starting anaconda: > > udevadm control --reload-rules Yeah I always forget that... OK so now we have a different outcome, but it's still something of an odd duck: It knows it's 9GiB, instead of 0 bytes. But it's still Unknown, and also even Unknown file system, so if we were to pretend this were a home array, I can't assign it the /home mount point. Unless it's saying it's unknown but really knows it's rootfs and hence not reusable?
Created attachment 1196053 [details] screenshot for comment 87
Created attachment 1196054 [details] storage.log c87
Created attachment 1196055 [details] program.log c87
Discussed during the 2016-08-29 blocker review meeting: [1] The decision to classify this bug as an AcceptedBlocker (Beta) was made as this bug is reproducible with clean disk images and violates the following Beta criteria: "When using the custom partitioning flow, the installer must be able to correctly interpret...any disk with a valid ms-dos or gpt disk label and partition table containing ext4 partitions, LVM and/or btrfs volumes, and/or software RAID arrays at RAID levels 0, 1 and 5 containing ext4 partitions" for live images. [1] https://meetbot.fedoraproject.org/fedora-blocker-review/2016-08-29/f25-blocker-review.2016-08-29-16.00.txt
Just tried updating my son's LVM-on-mdraid1 box to Fedora 24, using Fedora 24 Server on a USB Stick as I've gotten used to the Workstation installer not working for this case, and this time the Server installer isn't working either. The LVM PV partitions just show up under "unknown". I can run "vgscan", "vgchange -ay" and "vgdisplay" on another VT and all of the existing filesystems show up there but I can't get the installer to see them. Rescanning disks in the installer doesn't help. "rd.auto" on the kernel line doesn't help either. Is there any workaround I could try here or am I just going to have to skip F24? Repartitioning and losing data is not an option for me.
This is fine as an excercise for this bug, but if you want to update, simply start fedora-upgrade from inside of the existing installation or use recommended https://fedoraproject.org/wiki/DNF_system_upgrade - this should avoid anaconda/blivet/whatever installer.
In fact if you just run GNOME Software and go to the 'Updates' tab you should see a nice graphical 'Upgrade to Fedora 24' option.
Would that work from an F22 install too? I generally do fresh installs into an alternate set of filesystems, so I can fall back to the previous release if necessary, or refer to config files from the previous release etc. So I current have an up to date F23 and its predecessor F22 install available for upgrade. Normally I'd overwrite the F22 install with the F24 one but I suppose I could upgrade that instead. Is it not surprising that an F24 Server install can't see the volumes? Might the netinstall be any better?
"Is it not surprising that an F24 Server install can't see the volumes?" Sure, it's not expected. Without any logs, though, we can't tell what's going on at all (whether you're seeing the same case as cmurf, or something else). Yes, upgrading via dnf-plugin-upgrade works fine from F22 (don't recall if we enabled graphical upgrades from F22 or not).
(In reply to Paul Howarth from comment #92) > Just tried updating my son's LVM-on-mdraid1 box to Fedora 24, using Fedora > 24 Server on a USB Stick as I've gotten used to the Workstation installer > not working for this case, and this time the Server installer isn't working > either. The LVM PV partitions just show up under "unknown". > > I can run "vgscan", "vgchange -ay" and "vgdisplay" on another VT and all of > the existing filesystems show up there but I can't get the installer to see > them. Rescanning disks in the installer doesn't help. "rd.auto" on the > kernel line doesn't help either. Is there any workaround I could try here or > am I just going to have to skip F24? Repartitioning and losing data is not > an option for me. Most probably fixed by the attachment in: https://bugzilla.redhat.com/show_bug.cgi?id=1361611#c8
I have NEC EXPRESS 5800/R120B-2 with BIOS raid; I'm facing the same issue when I install F25 Alpha, F24,F23,F22. I have also tried Centos 7 and 7.2 with the same issue. Only F21 64bit and Centos 6.8 64bit can detect BIOS raid H.D. and can be installed on this legacy H.W. I guess it's the same Centos bug reported on this link https://bugs.centos.org/view.php?id=9813
Created attachment 1204630 [details] lspci Output
Comment on attachment 1204630 [details] lspci Output this is the output of "lspci -k" from Centos 6.8 running the below kernel version: Linux localhost.localdomain 2.6.32-642.3.1.el6.x86_64 #1 SMP Tue Jul 12 18:30:56 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
So, back around comments 66 through 90, I think we had at least zero'ed in on a specific case here: detection of existing software RAID sets in the live installer. I'm not sure Paul's or Fawzy's comments are about the same situation at all (Paul, at least, was clearly using a non-live installer image). To avoid confusion, can you two please file separate bugs if you're reporting different cases? Thanks.
Hi David, We looked through this bug at the 2016-09-26 Blocker-Review-Meeting and were not exactly sure on where this bug stands. Can you clarify as to what your understanding of this bug is and if you think any progress is being made? Thanks, Geoff M. - QA IRC: coremodule
There are a variety of things reported here, most notably: 1. failure to detect fwraid that uses DDF metadata - DDF metadata is not supported by either dmraid or mdadm, making it impossible for the installer to support it 2. failure to detect Intel imsm/isw fwraid in live media installations - read on There are some obsolete bits related to md in anaconda that are probably contributing to this: 1. setting ANACONDA udev env variable to prevent md udev rules from auto-assembling md arrays 2. deactivating all non-critical storage before & after running anaconda in the live environment Those are both anaconda changes. I'm going to proceed with those and then we'll see where it stands.
OK. I just have one more question: cmurf's reproducer definitely reads like he actually used *software* RAID, not any kind of firmware RAID. kparal's comment (66) is ambiguous, I cannot tell from his comment whether he used firmware or software RAID. Do you think 2. could affect software RAID as well as Intel firmware RAID? I've adjusted the summary to at least specify we're talking about the *live* installer here, not anything else. Other reporters: if what you're reporting is basically #1, I think the story here is that anaconda cannot do anything about it if the underlying tools don't support that metadata format, so basically what you need is a feature request for mdadm and/or dmraid. If what you're reporting is something else, please file a new bug (or find a closer duplicate for your case). Thanks!
My testing is mdadm RAID as created by the Fedora 24 installer. Fedora 25 Live installer shows the arrays' names in custom partitioning, but shows them as 0 bytes in size, they can't be reused only deleted, and it doesn't see the Fedora 24 install. Fedora 25 netinstall shows the arrays' names in custom partitioning, and their sizes, and sees the Fedora 24 installation, and I can reuse the arrays, i.e. I can assign the Fedora 24 home to /home.
(In reply to Adam Williamson from comment #104) > kparal's comment 66 is ambiguous, I cannot tell from his comment whether he used > firmware or software RAID. Sorry. I used software RAID, performed in a VM.
(In reply to Adam Williamson from comment #104) > firmware or software RAID. Do you think 2. could affect software RAID as > well as Intel firmware RAID? Yes, I think it could.
Upstream pull request: https://github.com/rhinstaller/anaconda/pull/812
So, the fix merged by dlehman does fix the software RAID case for me. To test, I did a software RAID install to a VM with two clean VirtIO disks with Fedora-Workstation-Live-x86_64-25-20160927.n.0.iso . Then I booted back to the same image, ran the installer again, and verified it behaves as described by cmurf and kparal: the RAID set is detected but shows in custom partitioning as 0 bytes in size under 'Unknown'. Then I rebooted and updated to an anaconda scratch build with dlehman's patches - http://koji.fedoraproject.org/koji/taskinfo?taskID=15834767 - and ran the installer again: now the partition on the RAID set is detected with its correct size and appears under 'Fedora 25' in custom partitioning. For the record, on my bare metal test system, current F25 nightlies cannot detect an Intel firmware RAID set *at all* with or without dlehman's patches. But this is clearly something different. I'll file a separate bug for that.
I have tried also with latest F25 netinstall CD without dlehman's patches ; and it can not detect my Intel firmware RAID set. I tried with F21 and upgraded it to F24 and F24 is working fine now on my Intel firmware RAID !! Is there a hope to get F25 anaconda detects my Intel firmware raid and install on my legacy servers?
The firmware RAID detection bug is https://bugzilla.redhat.com/show_bug.cgi?id=1379865 .
Re-assigning to anaconda as that's where the fix went, setting MODIFIED as the PR has been merged, and setting appropriate fixed-in version.
anaconda-25.20.4-1.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-067d26633e
Verified the fix in Beta-1.1.
anaconda-25.20.4-1.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.
Unfortunatelly the BUG stil persists, I have faced it when created RAD0 array on ASROck N68C-s UCC matherboard, RAID0 array of type nvidia_ceeiebdh I have tested it on Fedora 25 Workstation, and Fedora 25 Server edition. a few lines from traceback ...blivet/osinstall.py line 1175 in storage_initialize storate_reset() ...python3.5/threading.py line 862 in run self_target(*self_args, **self_kwargs) ...pyanaconda/threads.py line 251 in run threading.Thread.run(self, *args, **kwargs) gi.overrides.BlockDev.DMError. Failed to activate the RAID set 'nvidia_ceeiebdh'
That isn't at all the same bug as this. Please file a new bug.