Bug 1225184 - Live installer does not detect RAID devices
Summary: Live installer does not detect RAID devices
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: anaconda
Version: 25
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Anaconda Maintenance Team
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: https://fedoraproject.org/wiki/Common...
Depends On:
Blocks: F25BetaBlocker
TreeView+ depends on / blocked
 
Reported: 2015-05-26 18:50 UTC by Daniel
Modified: 2017-05-02 13:53 UTC (History)
34 users (show)

Fixed In Version: anaconda-25.20.4-1.fc25
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-10-07 03:34:21 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
var_live_raid_error_ws.tar.bz2 (26.54 KB, application/x-bzip)
2015-05-29 09:18 UTC, Daniel
no flags Details
proc_live_raid_error_ws.tar.bz2 (28.98 KB, application/x-bzip)
2015-05-29 09:19 UTC, Daniel
no flags Details
partitions /dev/sda-c (1.70 KB, text/plain)
2015-05-29 09:20 UTC, Daniel
no flags Details
etc_live_raid_error_ws.tar.bz2 (110 bytes, application/x-bzip)
2015-05-29 09:22 UTC, Daniel
no flags Details
mdadm_live_raid_error_ws.tar.bz2 (962 bytes, application/x-bzip)
2015-05-29 09:23 UTC, Daniel
no flags Details
rpm --query --all --qf "%-30{NAME} - %{VERSION}\n" | sort (55.99 KB, text/plain)
2015-05-29 09:25 UTC, Daniel
no flags Details
rpm -qa --queryformat='%{NAME}\n' | sort (18.78 KB, text/plain)
2015-05-29 09:26 UTC, Daniel
no flags Details
thinkpad x300 /var/log/boot.log (16.41 KB, text/plain)
2015-06-03 14:28 UTC, Daniel
no flags Details
thinkpad x300 dmesg (75.78 KB, text/plain)
2015-06-03 14:28 UTC, Daniel
no flags Details
Anaconda log files (110.18 KB, application/x-gzip)
2015-12-08 12:08 UTC, Terry Barnaby
no flags Details
Fedora 21 mdstat and mdadm.conf (576 bytes, text/plain)
2016-06-26 16:39 UTC, Erik M Jacobs
no flags Details
Fedora 21 fdisk -l output (2.33 KB, text/plain)
2016-06-26 16:39 UTC, Erik M Jacobs
no flags Details
Fedora 21 dmesg output (60.23 KB, text/plain)
2016-06-26 16:40 UTC, Erik M Jacobs
no flags Details
Fedora 24 dmesg output (64.38 KB, text/plain)
2016-06-26 16:41 UTC, Erik M Jacobs
no flags Details
Fedora 24 fdisk -l output (3.20 KB, text/plain)
2016-06-26 16:41 UTC, Erik M Jacobs
no flags Details
F24 Anaconda log (13.53 KB, text/plain)
2016-07-13 15:06 UTC, Erik J
no flags Details
F24 storage log (97.84 KB, text/plain)
2016-07-13 15:07 UTC, Erik J
no flags Details
F24 fdisk output (2.14 KB, text/plain)
2016-07-13 15:08 UTC, Erik J
no flags Details
F24 dmraid output (207 bytes, text/plain)
2016-07-13 15:08 UTC, Erik J
no flags Details
F24 smartctl output (1.62 KB, text/plain)
2016-07-13 15:09 UTC, Erik J
no flags Details
anaconda exception report (331.37 KB, text/plain)
2016-08-02 14:39 UTC, Erik J
no flags Details
anaconda.log (7.18 KB, text/plain)
2016-08-02 14:40 UTC, Erik J
no flags Details
storage.log (6.21 KB, text/plain)
2016-08-02 14:40 UTC, Erik J
no flags Details
storage.log 20160829 (123.81 KB, text/plain)
2016-08-29 18:24 UTC, Chris Murphy
no flags Details
program.log 20160829 (26.04 KB, text/plain)
2016-08-29 18:25 UTC, Chris Murphy
no flags Details
anaconda.log 20160829 (22.45 KB, text/plain)
2016-08-29 18:39 UTC, Chris Murphy
no flags Details
screenshot 20160829 (126.86 KB, image/png)
2016-08-29 18:40 UTC, Chris Murphy
no flags Details
program.log 20160830 (26.25 KB, text/plain)
2016-08-30 17:23 UTC, Chris Murphy
no flags Details
storage.log 20160830 (124.25 KB, text/plain)
2016-08-30 17:23 UTC, Chris Murphy
no flags Details
screenshot for comment 87 (133.51 KB, image/png)
2016-08-30 18:36 UTC, Chris Murphy
no flags Details
storage.log c87 (126.36 KB, text/plain)
2016-08-30 18:38 UTC, Chris Murphy
no flags Details
program.log c87 (26.25 KB, text/plain)
2016-08-30 18:38 UTC, Chris Murphy
no flags Details
lspci Output (8.91 KB, text/plain)
2016-09-25 20:57 UTC, Fawzy
no flags Details

Description Daniel 2015-05-26 18:50:44 UTC
Description of problem: 
Cannot install F22, since installer does not recognise my RAID devices from F21 installation.

Burned to DVD and verified image Fedora-Live-Workstation-x86_64-22-3.iso and booted from disk. After a very long boot time I did try to upgrade my F21 system. Installer shows for some seconds a "Sorry, ... problem ..." message but than shows normal install screens. The Installer sees all single partitions except my RAID device.

From the working F21 system this device looks as follows:
# mdadm -E /dev/sd[b-c]
/dev/sdb:
   MBR Magic : aa55
Partition[0] :   4294967295 sectors at            1 (type ee)
/dev/sdc:
   MBR Magic : aa55
Partition[0] :   4294967295 sectors at            1 (type ee)


# mdadm -E /dev/sd[b-c]1
/dev/sdb1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : b145e209:0ec81538:744622c4:c33ae860
           Name : bl4ckh0l4.s4y.no:opt_yachad00
  Creation Time : Fri Jan 16 16:05:19 2015
     Raid Level : raid0
   Raid Devices : 2

 Avail Dev Size : 5000290304 (2384.32 GiB 2560.15 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=0 sectors
          State : clean
    Device UUID : 7f1b3ea8:259fbe15:c7e547c6:9408f126

    Update Time : Fri Jan 16 16:05:19 2015
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : a69b8d5 - correct
         Events : 0

     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : b145e209:0ec81538:744622c4:c33ae860
           Name : bl4ckh0l4.s4y.no:opt_yachad00
  Creation Time : Fri Jan 16 16:05:19 2015
     Raid Level : raid0
   Raid Devices : 2

 Avail Dev Size : 5000290304 (2384.32 GiB 2560.15 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=0 sectors
          State : clean
    Device UUID : 6bd4a9ec:204bf024:33c224ad:45393e39

    Update Time : Fri Jan 16 16:05:19 2015
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 57312ada - correct
         Events : 0

     Chunk Size : 512K

   Device Role : Active device 1
   Array State : AA ('A' == active, '.' == missing, 'R' == replacing)

Comment 1 Daniel 2015-05-26 23:08:42 UTC
The installation process using the server edition with Fedora-Server-DVD-x86_64-22.iso works fine. RAID partitions will correctly be recognised.

Comment 2 Brian Lane 2015-05-27 16:02:24 UTC
If the Server is working and Live isn't then something must be missing from the kickstart used to create the Live images.

Comment 3 Kevin Fenzi 2015-05-27 19:27:21 UTC
Odd. I can't think of what this might be. 

Can you tell us more about the raid? Model/make/etc...

Could you attach a 'dmesg' from both f22 live and f22 install media boots?

Comment 4 Daniel 2015-05-29 09:17:20 UTC
Now, I did the following:
o used Server-DVD to install a system
o installed with Live-Workstation on a notebook without problems (no raid)
o installed on the server all the packages from WS installation from notebook
o changed boot target and some stuff (first of all 'dnf remove xorg-x11-drv-libinput-0.9.0-1.fc22.x86_64' since my mouse is nearly not usable under F22) and have now a shiny new F22 system! :-)

Repeated the live install and I've got the same error messages and no raid detected. Saved some stuff from this live session incl. the abrt report from /var/tmp/abrt/ and will attach it now.

Comment 5 Daniel 2015-05-29 09:18:30 UTC
Created attachment 1031757 [details]
var_live_raid_error_ws.tar.bz2

Comment 6 Daniel 2015-05-29 09:19:10 UTC
Created attachment 1031758 [details]
proc_live_raid_error_ws.tar.bz2

Comment 7 Daniel 2015-05-29 09:20:32 UTC
Created attachment 1031759 [details]
partitions /dev/sda-c

Comment 8 Daniel 2015-05-29 09:22:33 UTC
Created attachment 1031760 [details]
etc_live_raid_error_ws.tar.bz2

Comment 9 Daniel 2015-05-29 09:23:05 UTC
Created attachment 1031761 [details]
mdadm_live_raid_error_ws.tar.bz2

Comment 10 Daniel 2015-05-29 09:25:55 UTC
Created attachment 1031762 [details]
rpm --query --all  --qf "%-30{NAME} - %{VERSION}\n" | sort

Comment 11 Daniel 2015-05-29 09:26:34 UTC
Created attachment 1031763 [details]
rpm -qa --queryformat='%{NAME}\n' | sort

Comment 12 Kevin Fenzi 2015-05-30 17:51:09 UTC
ok, this is intel fw raid. I wonder if it's: 

https://bugzilla.redhat.com/show_bug.cgi?id=1219264

Comment 13 Daniel 2015-05-31 12:39:37 UTC
But, interestingly, why does the server edition not have the same bug? I've had this (software raid) configuration since F20 at least. And it always worked without problems.

Comment 14 Kevin Fenzi 2015-05-31 20:13:10 UTC
No idea, but you can see AdamW saw the same thing in that bug... server/install dvd worked, live didn't.

Comment 15 Andreas M. Kirchwitz 2015-06-01 13:10:25 UTC
With the final F22 Workstation installation image (x86, 64 bit) I've the same problem for my existing F21 installation based on regular software RAID-1 (md*).

When I boot the Workstation image (USB stick) and select "Install to Harddisk",
first some mysterious crash occurs (Oops...) but eventually the Installer starts up. Unfortunately, when selecting the destination devices, no software RAID devices (md*) from my existing F21 installation will show up. So there's nothing I can install F22 on. There's no hardware-based RAID involved or any motherboard RAID functions.

Works fine with the Server Image.

Looks similar to bug #1225671

Comment 16 Daniel 2015-06-03 14:26:37 UTC
Just tried another install on a Thinkpad X300 with Fedora-Live-Workstation. This time and on that specific hardware it stops with an error message "Oh No Something went wrong. The system encountered a problem and can't recover.".

Applying the same procedure as layed out in "comment 4" I could finally install F22 workstation.

So far the workstation install worked for me on a single hardware (Thinkpad T440s) out of the box, On all outher notebooks and pc's it just failed. ;-\

On that Thinkpad X300 there is no information in /var/tmp/abrt. I'll attach the /var/lob/boot.log and dmesg.

Comment 17 Daniel 2015-06-03 14:28:20 UTC
Created attachment 1034360 [details]
thinkpad x300 /var/log/boot.log

Comment 18 Daniel 2015-06-03 14:28:57 UTC
Created attachment 1034361 [details]
thinkpad x300 dmesg

Comment 19 Andreas M. Kirchwitz 2015-11-05 15:33:11 UTC
Essentially, this problem has not been fixed for Fedora 23. Although RAID devices no longer cause crashes in Anaconda and are somehow recognized, they are flagged es "Unknown" (unknown size, unknown label, unknown formatting etc) and cannot be used during the Installation process.

Several bug reports where filed for this issue. All ignored. Does that mean, Software RAID is no longer supported officially and users have to switch distribution?

Comment 20 Daniel 2015-11-05 20:03:53 UTC
Yep, have seen it in F23. Workstation image crashed, and i've used the server edition to update systems. It is not just RAID. Encrypted partitions are not liked as well. :-\

Comment 21 Adam Pribyl 2015-11-29 17:50:57 UTC
I can not boot F22 kernels on mdadm raid device with LVM over it. I am dropped to dracut shell. blkid shows correctly all the devices, but they are not assembled, the of course the LVM which is over the raid fails too
+ command -v lvm
+ lvm pvdisplay
+ lvm vgdisplay
+ lvm lvdisplay
+ command -v dmsetup
+ dmsetup ls --tree
No devices found
+ cat /proc/mdstat
Personalities :
unused devices: <none>

As I can reproduce this on every boot with F22 kernel (F21 kernel works!) I can test whatever will make sense. 

I'd like to point out, that grub.cfg generated by grub2-mkconfig fails badly too, but this may not be related.

Comment 22 Terry Barnaby 2015-12-07 20:34:24 UTC
Just been bitten by this bug for Fedora 23. It is ridiculous that this has not been fixed since Fedora 22 it is pretty basic required functionality and shows the very limited testing that Anaconda/install has had.

Comment 23 Chris Lumens 2015-12-07 21:27:28 UTC
Cool, thanks for that!

Comment 24 Terry Barnaby 2015-12-08 12:05:01 UTC
Sorry, spent almost a day trying to work around this and was quite frustrated.
I also tried Fedora-Server-DVD-x86_64-23.iso and that is ok (but not very suitable for our use). The contents of this is significantly different to the workstation live images, not sure why this should be so ?
Have had a play with trying to debug what is happening, but haven't the knowledge or time to delve deeply. I have run the "liveinst" script from a GUI terminal and
had a look at the files in /tmp.

Some things:

1. The raid devices are not assembled/started when anaconda is running. In fact
  the raid arrays are stopped when "liveinst" is run (If I start up the arrays
  manually before).

2. I see messages like:
   DEBUG blivet: MDRaidArrayDevice.readCurrentSize: path: /dev/md/4 ; exists: True ; sysfsPath:  ;
   DEBUG blivet: existing RAID raid1 size == 0 B
   in /tmp/storage.log
   There is no /dev/md/4 as the raid devices is shutdown.

3. If I run "anaconda" directly I get what looks like a "future" anaconda
  with lots of warning messages about being development software etc. However
  if I proceed to the manual disk partitioning page it does see the raid
  partitions with the correct size. I haven't done further with this.

I will attach the /tmp/*.log files in case they are useful.

Comment 25 Terry Barnaby 2015-12-08 12:08:21 UTC
Created attachment 1103567 [details]
Anaconda log files

Comment 26 Adam Pribyl 2015-12-08 13:00:14 UTC
For my case from Comment 21 I have "resolution" here: https://bugzilla.redhat.com/show_bug.cgi?id=1286464

Comment 27 Andreas M. Kirchwitz 2015-12-09 16:01:14 UTC
The Fedora Server ISO image can be used as a workaround.

Either perform a "minimal install" (Server) and once the system is installed use "dnf" to switch to the Workstation distribution. Or select the Workstation installation within the Server installer. However, both ways require a good Internet connection as they both download all packages via network. (Unfortunately, the Server ISO doesn't allow to use a local Workstation ISO image file as source.)

Furthermore, the packages are a little bit different to a real Workstation installation. Some additional work is required after installation to finally have a proper Workstation environment.

Anyway, this bug should be fixed. It's strange that Fedora fails so badly on RAID devices or encrypted volumes. It worked well up to F21. Question is if there are any intentions to fix it, and if not, will upcoming Red Hat releases (which are based on Fedora) also stop support for RAID and encrypted devices?

Comment 28 Matthew Miller 2016-02-02 09:17:08 UTC
Updating based on comments that this still occurs on F23.

Comment 29 Matthew Miller 2016-02-02 09:18:53 UTC
Based on the comment that this works with the Server installer, I expect that using the Workstation netinst ISO should serve as another workaround. Download from https://getfedora.org/en/workstation/download/ -- look in the right column -- or direct link https://download.fedoraproject.org/pub/fedora/linux/releases/23/Workstation/x86_64/iso/Fedora-Workstation-netinst-x86_64-23.iso

Can someone confirm?

Comment 30 Rohit Shriwas 2016-03-04 08:03:11 UTC
Matthew, the Workstation netinst ISO correctly detects the software RAID and offers the mapped block devices as targets for installation.

Comment 31 silvertip257 2016-03-09 01:09:28 UTC
By utilizing the F23 netinstall image, I was able to perform an LVM on Software RAID install.

Comment 32 Terry Barnaby 2016-03-09 09:21:42 UTC
So 8 months on and still no fix for this ?

Comment 33 Erik J 2016-04-27 00:07:38 UTC
I encountered this problem trying to install F23 on a fake raid system (AMD SB950 with mirrored disks). I tried installing the F23 scientific spin, the F23 workstation spin and the F23 net install spin. None of these would recognize the raid drive. No drives except the live USB were visible for installation. I tried using wipefs to clear the hard drives and recreate the raid logical volume, but that was a no go as well.

FWIW: the newly released Kubuntu 16.04 live install iso does recognize the raid and installs OK so I know the raid volume and the disks are not the problem.

In the end I had to delete the raid logical volume and install F23 on a single drive, which worked fine.

I hope this problem can get some attention, as I would like to stick with Fedora and eventually migrate back to a mirrored raid system.

Thanks for any help you can give me.

Comment 34 silvertip257 2016-05-03 01:55:01 UTC
Erik J:
Comments 28 through 31 provide a workaround.
The normal workstation ISO does not work for softraid (mdadm) disk layouts, but the netinstall ISO does (I tried the workstation one and had success). Use this method if you want to stick with Fedora.

Terry Barnaby:
I wouldn't expect a fix until the next release (F24).
Since F24 final release is expected in June 2016 I would question a fix will make that release. https://fedoraproject.org/wiki/Releases/24/Schedule
I don't see any mention of "raid" in the common bugs page (not that its absence necessarily means anything). https://fedoraproject.org/wiki/Common_F24_bugs

Comment 35 Erik J 2016-05-03 04:33:15 UTC
silvertip257:
Thanks for the reply. Please note that I did try the netinstall ISO and it *did not* recognize the raid, which is why I posted the comment. I read the entire thread carefully and tried the workarounds mentioned. When the netinstall did not work I wanted to bring it to people's attention. I am happy to try other options or perform specific tests if it will help. Thanks.

Comment 36 Erik J 2016-06-25 05:37:09 UTC
This bug still exists in F24. Neither Workstation ISO, nor Workstation netinst recognize softraid. I have read the comments and work-arounds in the thread, but they do not work for my HW.

Comment 37 Erik J 2016-06-25 16:40:27 UTC
A clarification, which I think explains why the work-around described above does not work in my case: I am using firmware raid, which is not the same as mdadm. The problem is that anaconda does not detect the firmware raid. I am sure the work-around is fine for mdadm, but it does not work for firmware raid. Sorry for any confusion I may have caused.

Comment 38 Erik M Jacobs 2016-06-26 16:35:43 UTC
I am also noticing this problem in F24.

I have an Intel Rapid Storage **HARDWARE** raid that is mirroring my two devices. In Fedora 21 the RAID was properly detected and assembled and I was able to create a partition on the RAID-ed device for installation of Fedora.

In Fedora 24 workstation live the device is not properly detected at all. It's really quite a mess. The following commands were all issued on Fedora 24 in a live session:

[root@localhost ~]# cat /proc/mdstat
Personalities : 
md127 : inactive sda[0](S)
      3028 blocks super external:imsm
       
unused devices: <none>

[root@localhost ~]# mdadm --detail-platform
       Platform : Intel(R) Rapid Storage Technology
        Version : 13.0.0.2075
    RAID Levels : raid0 raid1 raid10 raid5
    Chunk Sizes : 4k 8k 16k 32k 64k 128k
    2TB volumes : supported
      2TB disks : supported
      Max Disks : 6
    Max Volumes : 2 per array, 4 per controller
 I/O Controller : /sys/devices/pci0000:00/0000:00:1f.2 (SATA)
          Port0 : /dev/sda (WD-WCC4NKTRPZZ0)
          Port2 : - non-disk device (HP DVD Writer 1160d) -
          Port1 : /dev/sdb (WD-WCC4NRNX77SK)
          Port3 : - no device attached -

[root@localhost ~]# man mdadm
[root@localhost ~]# mdadm -E /dev/sda1
mdadm: cannot open /dev/sda1: No such file or directory
[root@localhost ~]# mdadm -E /dev/sda
/dev/sda:
          Magic : Intel Raid ISM Cfg Sig.
        Version : 1.1.00
    Orig Family : ab616185
         Family : ab616185
     Generation : 004f8499
     Attributes : All supported
           UUID : e97d24bf:d4c05d9f:899652c4:01270fbd
       Checksum : b3cddd5e correct
    MPB Sectors : 1
          Disks : 2
   RAID Devices : 1

  Disk00 Serial : WD-WCC4NKTRPZZ0
          State : active
             Id : 00000000
    Usable Size : 3907023112 (1863.01 GiB 2000.40 GB)

[Main]:
           UUID : dac43f62:7f66c7ad:5a0238af:fcd2ff4f
     RAID Level : 1
        Members : 2
          Slots : [UU]
    Failed disk : none
      This Slot : 0
     Array Size : 3907022848 (1863.01 GiB 2000.40 GB)
   Per Dev Size : 3907023112 (1863.01 GiB 2000.40 GB)
  Sector Offset : 0
    Num Stripes : 15261808
     Chunk Size : 64 KiB
       Reserved : 0
  Migrate State : idle
      Map State : normal
    Dirty State : clean

  Disk01 Serial : WD-WCC4NRNX77SK
          State : active
             Id : 00000001
    Usable Size : 3907023112 (1863.01 GiB 2000.40 GB)
[root@localhost ~]# 
[root@localhost ~]# 
[root@localhost ~]# mdadm -E /dev/sdb
/dev/sdb:
          Magic : Intel Raid ISM Cfg Sig.
        Version : 1.1.00
    Orig Family : ab616185
         Family : ab616185
     Generation : 004f8499
     Attributes : All supported
           UUID : e97d24bf:d4c05d9f:899652c4:01270fbd
       Checksum : b3cddd5e correct
    MPB Sectors : 1
          Disks : 2
   RAID Devices : 1

  Disk01 Serial : WD-WCC4NRNX77SK
          State : active
             Id : 00000001
    Usable Size : 3907023112 (1863.01 GiB 2000.40 GB)

[Main]:
           UUID : dac43f62:7f66c7ad:5a0238af:fcd2ff4f
     RAID Level : 1
        Members : 2
          Slots : [UU]
    Failed disk : none
      This Slot : 1
     Array Size : 3907022848 (1863.01 GiB 2000.40 GB)
   Per Dev Size : 3907023112 (1863.01 GiB 2000.40 GB)
  Sector Offset : 0
    Num Stripes : 15261808
     Chunk Size : 64 KiB
       Reserved : 0
  Migrate State : idle
      Map State : normal
    Dirty State : clean

  Disk00 Serial : WD-WCC4NKTRPZZ0
          State : active
             Id : 00000000
    Usable Size : 3907023112 (1863.01 GiB 2000.40 GB)
[root@localhost ~]# 
[root@localhost ~]# 
[root@localhost ~]# mdadm -E /dev/sdb1
/dev/sdb1:
   MBR Magic : aa55
Partition[0] :   1836016416 sectors at   1936269394 (type 4f)
Partition[1] :    544437093 sectors at   1917848077 (type 73)
Partition[2] :    544175136 sectors at   1818575915 (type 2b)
Partition[3] :        54974 sectors at   2844524554 (type 61)

I will submit attachments for various other things. Supposedly there is a workaround but I could not understand it from this thread.

Also note that:

https://bugzilla.redhat.com/show_bug.cgi?id=1219264
https://bugzilla.redhat.com/show_bug.cgi?id=1172426

both appear to be related.

Comment 39 Erik M Jacobs 2016-06-26 16:39:15 UTC
Created attachment 1172604 [details]
Fedora 21 mdstat and mdadm.conf

I did not create the mdadm.conf. I am assuming the installer created it when I installed F21.

Comment 40 Erik M Jacobs 2016-06-26 16:39:47 UTC
Created attachment 1172605 [details]
Fedora 21 fdisk -l output

Comment 41 Erik M Jacobs 2016-06-26 16:40:32 UTC
Created attachment 1172606 [details]
Fedora 21 dmesg output

Comment 42 Erik M Jacobs 2016-06-26 16:41:14 UTC
Created attachment 1172607 [details]
Fedora 24 dmesg output

Comment 43 Erik M Jacobs 2016-06-26 16:41:56 UTC
Created attachment 1172608 [details]
Fedora 24 fdisk -l output

Comment 44 Erik M Jacobs 2016-06-26 16:43:58 UTC
Note that while the RAID device is not detected, one of the member disks is detected and its partitions are shown:

[root@localhost ~]# fdisk -l /dev/sdb
Disk /dev/sdb: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x0001f847

Device     Boot      Start        End    Sectors   Size Id Type
/dev/sdb1  *          2048     206847     204800   100M  7 HPFS/NTFS/exFAT
/dev/sdb2           206848 2977521663 2977314816   1.4T  7 HPFS/NTFS/exFAT
/dev/sdb3       2977521664 3790045183  812523520 387.5G 8e Linux LVM

Additionally, the PV/LV/VG information is available, too:

[root@localhost ~]# pvs
  PV         VG     Fmt  Attr PSize   PFree
  /dev/sdb3  fedora lvm2 a--  387.44g 4.00m
[root@localhost ~]# lvs
  LV   VG     Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  home fedora -wi-a----- 279.40g                                                    
  root fedora -wi-a-----  93.13g                                                    
  swap fedora -wi-ao----  14.90g                                                    
[root@localhost ~]# vgs
  VG     #PV #LV #SN Attr   VSize   VFree
  fedora   1   3   0 wz--n- 387.44g 4.00m

[root@localhost ~]# vgdisplay
  --- Volume group ---
  VG Name               fedora
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  4
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                3
  Open LV               1
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               387.44 GiB
  PE Size               4.00 MiB
  Total PE              99184
  Alloc PE / Size       99183 / 387.43 GiB
  Free  PE / Size       1 / 4.00 MiB
  VG UUID               wb4KFn-nxCJ-beqy-nXjA-u2N0-Gnv7-dfRCNB
   
[root@localhost ~]# pvdisplay
  --- Physical volume ---
  PV Name               /dev/sdb3
  VG Name               fedora
  PV Size               387.44 GiB / not usable 4.00 MiB
  Allocatable           yes 
  PE Size               4.00 MiB
  Total PE              99184
  Free PE               1
  Allocated PE          99183
  PV UUID               J1diTf-CZLP-D8dW-TGuT-LOzR-y3s8-8oQs0I

[root@localhost ~]# lvdisplay
  --- Logical volume ---
  LV Path                /dev/fedora/root
  LV Name                root
  VG Name                fedora
  LV UUID                BeGdLK-r9iu-kZgf-RdpM-vRFM-iQS3-E2OhK2
  LV Write Access        read/write
  LV Creation host, time localhost, 2015-01-11 21:29:11 -0500
  LV Status              available
  # open                 0
  LV Size                93.13 GiB
  Current LE             23842
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0
   
  --- Logical volume ---
  LV Path                /dev/fedora/home
  LV Name                home
  VG Name                fedora
  LV UUID                19Epia-1Rj6-ulxP-Lu25-0UOS-tBDu-woicRw
  LV Write Access        read/write
  LV Creation host, time localhost, 2015-01-11 21:29:16 -0500
  LV Status              available
  # open                 0
  LV Size                279.40 GiB
  Current LE             71526
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:1
   
  --- Logical volume ---
  LV Path                /dev/fedora/swap
  LV Name                swap
  VG Name                fedora
  LV UUID                P7s3Y6-sQkW-Gwz3-4Qas-0Fjx-7gOZ-HRFnLL
  LV Write Access        read/write
  LV Creation host, time localhost, 2015-01-11 21:29:28 -0500
  LV Status              available
  # open                 2
  LV Size                14.90 GiB
  Current LE             3815
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:2


But **none** of these devices are shown as being available for the installer to use.

Comment 45 Erik M Jacobs 2016-07-04 19:10:47 UTC
So, I think the issue actually lies outside of Anaconda in my particular case.

By adding "rd.auto" to the startup of the live image, the RAID array was properly assembled and then available for use as an installation target. F24 then installed with no issue.

Comment 46 Erik J 2016-07-13 15:04:31 UTC
Hello again. I have tried the rd.auto method that Erik Jacobs mentions above, as well as rd.dm=1, but am having no luck. I found 2 other bug reports that are similar, but without any resolution or workarounds that work for me:

https://bugzilla.redhat.com/show_bug.cgi?id=1219264
https://bugzilla.redhat.com/show_bug.cgi?id=1225618

I have tried just about everything imaginable, including wiping the disks with nwipe and creating a fresh raid set. I have tried installing Fedora Workstation, Server, and tried netinst, but still no luck. I have uploaded various outputs from my install attempts below. Blivet complains that my disks do not appear to be part of an array, but later finds the raid device. the raid device is detected using dmraid as well. I am curious if the blivet warning is because the devices show up as "mirror" for RAID-1, which is not quite the same as part of an array as in RAID-5 or 10.

If I install gnome-disk-utility onto the live version it correctly detects the RAID.

Please point me in the right direction if this is not the right component under which this should be reported or if I should submit a new bug report. I am happy to help with any other tests you would like me to run to debug this.

My HW is using the AMD SB 950 chipset, which evidently is using a Promise FastTrack firmware RAID controller which is enabled in the BIOS. The RAID functionality works, as I have previously installed Linux Mint and Kubuntu on it, so it is likely not a problem with the RAID set. The install has failed for both F23 and F24 (works fine for individual disk installs, though). I have not tried any previous versions, although the 1219264 bug report seems to indicate that RAID sets were correctly detected previously for F20.

Thanks for any help or guidance you can give me.
Erik J

Comment 47 Erik J 2016-07-13 15:06:50 UTC
Created attachment 1179314 [details]
F24 Anaconda log

Comment 48 Erik J 2016-07-13 15:07:40 UTC
Created attachment 1179315 [details]
F24 storage log

Comment 49 Erik J 2016-07-13 15:08:20 UTC
Created attachment 1179317 [details]
F24 fdisk output

Comment 50 Erik J 2016-07-13 15:08:58 UTC
Created attachment 1179318 [details]
F24 dmraid output

Comment 51 Erik J 2016-07-13 15:09:34 UTC
Created attachment 1179319 [details]
F24 smartctl output

Comment 52 David Lehman 2016-07-19 16:00:50 UTC
vpodzime, do you know why libblockdev doesn't report any arrays for the promise fasttrack members in comment 46? It seems like dmraid sees them, but libblockdev's call to libdmraid does not.

Comment 53 Erik J 2016-07-19 16:40:38 UTC
Hi David and vpodzime,

I had a chance to debug this last weekend. Unfortunately, my notes are at home, so I will have to update this post later tonight with more details. I tracked everything down to a particular routine in blivet where it tries to add the array but fails because it does not return a valid device from getDeviceByName. Here is the pertinent output in storage.log where the problem occurs (see below). I am happy to help you troubleshoot this if you can give me a little guidance (my expertise is in signal processing and scientific sw). I already tried hacking a bit in blivet to try to force it to load the array into the device tree, but got stuck later on when it wanted the format type and some other info that was missing which caused blivet to bail. Thanks, Erik J.

09:04:39,320 INFO blivet: scanning pdc_iffcgcja (/sys/devices/virtual/block/dm-0)...
09:04:39,320 DEBUG blivet:        DeviceTree.getDeviceByName: name: pdc_iffcgcja ; hidden: False ; incomplete: False ;
09:04:39,321 DEBUG blivet:        DeviceTree.getDeviceByName returned None
09:04:39,321 INFO blivet: pdc_iffcgcja is a device-mapper device
09:04:39,321 DEBUG blivet:       Populator.addUdevDMDevice: name: pdc_iffcgcja ;
09:04:39,326 DEBUG blivet:          DeviceTree.getDeviceByName: name: sda ; hidden: False ; incomplete: False ;
09:04:39,327 DEBUG blivet:          DeviceTree.getDeviceByName returned existing 465.76 GiB disk sda (0) with existing dmraidmember
09:04:39,328 DEBUG blivet:          DeviceTree.getDeviceByName: name: sdb ; hidden: False ; incomplete: False ;
09:04:39,329 DEBUG blivet:          DeviceTree.getDeviceByName returned existing 465.76 GiB disk sdb (6) with existing dmraidmember
09:04:39,329 DEBUG blivet:         DeviceTree.getDeviceByName: name: pdc_iffcgcja ; hidden: False ; incomplete: False ;
09:04:39,330 DEBUG blivet:         DeviceTree.getDeviceByName returned None
09:04:39,330 DEBUG blivet: lvm filter: adding pdc_iffcgcja to the reject list
09:04:39,330 WARN blivet: ignoring dm device pdc_iffcgcja
09:04:39,330 DEBUG blivet: no device obtained for pdc_iffcgcja

Comment 54 Erik J 2016-07-20 18:43:17 UTC
A brief update:

In blivet:populator.py the method addUdevDevice is called to add the device to the tree (this is where the log note "scanning pdc_iffcgcja" is generated). In the method it calls self.getDeviceByName to get the device, which returns "None". Further down in the method it goes through a long if-elif statement to either lookup or create the device:

        #
        # The first step is to either look up or create the device
        #
        device_added = True
        if device:
            device_added = False
        elif udev.device_is_loop(info):
            log.info("%s is a loop device", name)
            device = self.addUdevLoopDevice(info)
        elif udev.device_is_dm_mpath(info) and \
             not udev.device_is_dm_partition(info):
            log.info("%s is a multipath device", name)
            device = self.addUdevMultiPathDevice(info)
        elif udev.device_is_dm_lvm(info):
            log.info("%s is an lvm logical volume", name)
            device = self.addUdevLVDevice(info)
        elif udev.device_is_dm(info):
            log.info("%s is a device-mapper device", name)
            device = self.addUdevDMDevice(info)
        elif udev.device_is_md(info) and not udev.device_get_md_container(info):
            log.info("%s is an md device", name)
            device = self.addUdevMDDevice(info)
        elif udev.device_is_cdrom(info):
            log.info("%s is a cdrom", name)
            device = self.addUdevOpticalDevice(info)
        elif udev.device_is_disk(info):
            device = self.addUdevDiskDevice(info)
        elif udev.device_is_partition(info):
            log.info("%s is a partition", name)
            device = self.addUdevPartitionDevice(info)
        else:
            log.error("Unknown block device type for: %s", name)
            return

        if not device:
            log.debug("no device obtained for %s", name)
            return

The device-mapper branch is taken and the method addUdevDMDevice is called. This returns "none" for the device so the original addUdevDevice method returns no device.

The addUdevDMDevice method calls self.getDeviceByName to get the device which, as before, returns "None" (it simply compares the name "pdc_iffcgcja" to all the devices already in the _devices[] array and no match is found). Later in the method we reach this code snippet:

        # if we get here, we found all of the slave devices and
        # something must be wrong -- if all of the slaves are in
        # the tree, this device should be as well
        if device is None:
            lvm.lvm_cc_addFilterRejectRegexp(name)
            log.warning("ignoring dm device %s", name)

        return device

The device is still "None" so no device is ever created or loaded and the method returns "None", so no further info on this device is made available to anaconda.

I don't know enough about how all the udev device things work in blivet, but my guess is that addUdevDMDevice should create the device and add it to the tree with the appropriate meta-data.

As I mentioned before, I am happy to help trouble shoot this if you can give me guidance on what to test.

Thanks, Erik J.

Comment 55 Vratislav Podzimek 2016-07-28 09:37:48 UTC
(In reply to David Lehman from comment #52)
> vpodzime, do you know why libblockdev doesn't report any arrays for the
> promise fasttrack members in comment 46? It seems like dmraid sees them, but
> libblockdev's call to libdmraid does not.

I have no idea. I'd need the particular piece of HW and do some step-by-step debugging to see what's going on.

Comment 56 Vratislav Podzimek 2016-07-28 09:44:35 UTC
Erik, could you please try to compare the results of the following?

1) install python-pyblock
2) run 'sudo python' and:
   >>> import block
   >>> block.getRaidSetFromRelatedMem(name=NAME_OF_ONE_RAID_MEMBER)

3) run 'sudo python3'
   >>> from gi.repository import BlockDev; BlockDev.init()
   >>> BlockDev.dm.get_member_raid_sets(name=NAME_OF_ONE_RAID_MEMBER)

libblockdev's get_member_raid_sets() function was written based on the pyblock's codebase so I'd like to see if they give the same results. Thanks!

Comment 57 Erik J 2016-07-28 18:12:20 UTC
Hi Vratislav,

Here is the output you requested. First a dmraid output so you can see the devices and names:

[root@shadowfax-amd-single erikj]# dmraid -r
/dev/sdb: pdc, "pdc_bbcbddjca", mirror, ok, 976562432 sectors, data@ 0
/dev/sda: pdc, "pdc_bbcbddjca", mirror, ok, 976562432 sectors, data@ 0

Output from python 2.7.12 and block:

[erikj@shadowfax-amd-single ~]$ sudo python
Python 2.7.12 (default, Jul 18 2016, 10:55:51) 
[GCC 6.1.1 20160621 (Red Hat 6.1.1-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import block
>>> block.getRaidSetFromRelatedMem(name="/dev/sda")
[]
>>> block.getRaidSetFromRelatedMem(name="/dev/sdb")
[]
>>> block.getRaidSetFromRelatedMem(name="pdc_bbcbddjca")
[]
>>> 

Output from python 3 and BlockDev:

[erikj@shadowfax-amd-single ~]$ sudo python3
Python 3.5.1 (default, Jul 10 2016, 20:36:01) 
[GCC 6.1.1 20160621 (Red Hat 6.1.1-3)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from gi.repository import BlockDev; BlockDev.init()
__main__:1: PyGIWarning: BlockDev was imported without specifying a version first. Use gi.require_version('BlockDev', '1.0') before import to ensure that the right version gets loaded.
True
>>> BlockDev.dm.get_member_raid_sets(name="/dev/sda")
[]
>>> BlockDev.dm.get_member_raid_sets(name="/dev/sdb")
[]
>>> BlockDev.dm.get_member_raid_sets(name="pdc_bbcbddjca")
[]
>>> 

In both cases they return nothing. I am not sure if I am using the correct format for the device names, please confirm.

Thanks,
Erik

Comment 58 Vratislav Podzimek 2016-07-29 07:39:47 UTC
Could you please also try just "sda" and "sdb"?

Comment 59 Erik J 2016-07-29 10:51:55 UTC
Sorry, don't know why I didn't think of that. OK, now we have output:

Python 2/7 and block:

>>> block.getRaidSetFromRelatedMem(name="sda")
[<block.device.RaidSet instance at 0x7faad89f0560>]
>>> block.getRaidSetFromRelatedMem(name="sdb")
[<block.device.RaidSet instance at 0x7faad89f0638>]

Python 3 and BlockDev:

>>> BlockDev.dm.get_member_raid_sets(name="sda")
['pdc_bbcbddjca']
>>> BlockDev.dm.get_member_raid_sets(name="sdb")
['pdc_bbcbddjca']

Comment 60 Anders Blomdell 2016-07-29 15:48:21 UTC
Maybe related: https://bugzilla.redhat.com/show_bug.cgi?id=1361611

At least your debugging put me on track on where to look, so kudos for that.

Comment 61 Vratislav Podzimek 2016-08-02 06:22:13 UTC
(In reply to Erik J from comment #59)
> Sorry, don't know why I didn't think of that. OK, now we have output:
> 
> Python 2/7 and block:
> 
> >>> block.getRaidSetFromRelatedMem(name="sda")
> [<block.device.RaidSet instance at 0x7faad89f0560>]
> >>> block.getRaidSetFromRelatedMem(name="sdb")
> [<block.device.RaidSet instance at 0x7faad89f0638>]
> 
> Python 3 and BlockDev:
> 
> >>> BlockDev.dm.get_member_raid_sets(name="sda")
> ['pdc_bbcbddjca']
> >>> BlockDev.dm.get_member_raid_sets(name="sdb")
> ['pdc_bbcbddjca']

Thanks! Looks like it could possibly be this stupid mistake:
https://github.com/rhinstaller/blivet/pull/473

Comment 62 Erik J 2016-08-02 14:37:05 UTC
Hi Vratislav,

I made this change to populator.py per your github link:

# diff populator.py populator.py~
1256c1256
<         rs_names = blockdev.dm.get_member_raid_sets(name, uuid, major, minor)
---
>         rs_names = blockdev.dm.get_member_raid_sets(uuid, name, major, minor)

Then I ran anaconda, which aborted with the following error:

anaconda 24.13.7-1 exception report
Traceback (most recent call first):
  File "/usr/lib64/python3.5/site-packages/gi/overrides/BlockDev.py", line 429, in wrapped
    raise transform[1](msg)
  File "/usr/lib/python3.5/site-packages/blivet/populator.py", line 1269, in handleUdevDMRaidMemberFormat
    blockdev.dm.activate_raid_set(rs_name)
  File "/usr/lib/python3.5/site-packages/blivet/populator.py", line 1487, in handleUdevDeviceFormat
    self.handleUdevDMRaidMemberFormat(info, device)
  File "/usr/lib/python3.5/site-packages/blivet/populator.py", line 773, in addUdevDevice
    self.handleUdevDeviceFormat(info, device)
  File "/usr/lib/python3.5/site-packages/blivet/populator.py", line 1701, in _populate
    self.addUdevDevice(dev)
  File "/usr/lib/python3.5/site-packages/blivet/populator.py", line 1638, in populate
    self._populate()
  File "/usr/lib/python3.5/site-packages/blivet/devicetree.py", line 555, in populate
    self._populator.populate(cleanupOnly=cleanupOnly)
  File "/usr/lib/python3.5/site-packages/blivet/blivet.py", line 279, in reset
    self.devicetree.populate(cleanupOnly=cleanupOnly)
  File "/usr/lib/python3.5/site-packages/blivet/osinstall.py", line 1157, in storageInitialize
    storage.reset()
  File "/usr/lib64/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib64/python3.5/site-packages/pyanaconda/threads.py", line 253, in run
    threading.Thread.run(self, *args, **kwargs)
gi.overrides.BlockDev.DMError: Failed to activate the RAID set 'pdc_bbcbddjca'


I have not had time to try to troubleshoot this yet.

I am attaching the anaconda crash output, anaconda.log and storage.log below.

Thanks,
Erik J

Comment 63 Erik J 2016-08-02 14:39:11 UTC
Created attachment 1186853 [details]
anaconda exception report

Comment 64 Erik J 2016-08-02 14:40:13 UTC
Created attachment 1186854 [details]
anaconda.log

Comment 65 Erik J 2016-08-02 14:40:46 UTC
Created attachment 1186855 [details]
storage.log

Comment 66 Kamil Páral 2016-08-29 16:50:32 UTC
This is still an issue with F25 Alpha Workstation Live. Reproducer:
* Install F24, standard partitions, onto two disks (raid 0). I used Workstation Live for that.
* Boot the system to verify it works.
* Boot F25 Workstation Live, go to custom partition, see that partitions are listed as unknown and cannot be used.

Setting a better component (blivet, since the patch went into blivet). Proposing as a Beta blocker:
"When using the custom partitioning flow, the installer must be able to: 
 Correctly interpret, and modify as described below, any disk with a valid ms-dos or gpt disk label and partition table containing ext4 partitions, LVM and/or btrfs volumes, and/or software RAID arrays at RAID levels 0, 1 and 5 containing ext4 partitions "

Comment 67 Adam Williamson 2016-08-29 17:36:17 UTC
As Kevin mentioned last year, this and https://bugzilla.redhat.com/show_bug.cgi?id=1219264 may well be the same , and I believe that for me involved one of the disks the RAID set was built out of having stale LVM metadata on it. I'd suggest checking the installer logs to see if it seems to be finding LVM volumes unexpectedly, and maybe running 'lvdisplay' and 'vgdisplay' and friends...

Comment 68 David Lehman 2016-08-29 17:37:59 UTC
(In reply to Kamil Páral from comment #66)
> This is still an issue with F25 Alpha Workstation Live. Reproducer:

Can you attach the logs?

Comment 69 Chris Murphy 2016-08-29 18:24:07 UTC
Created attachment 1195453 [details]
storage.log 20160829

Just like in kparal's comment 66. What I did was:
1. create two new qcow2 files
2. boot f24 installer, and do a single root mount point, XFS on mdadm raid0 installation, nothing else. It boots after installation fine.
3. boot f25 installer, and I get an "unknown" item in manual partitioning, under which is a 0byte "root" unknown item.

Comment 70 Chris Murphy 2016-08-29 18:25:30 UTC
Created attachment 1195454 [details]
program.log 20160829

program.log that goes with storage.log above

Comment 71 Chris Murphy 2016-08-29 18:39:17 UTC
Created attachment 1195458 [details]
anaconda.log 20160829

anaconda.log that goes with comment 69 and 70

Comment 72 Chris Murphy 2016-08-29 18:40:21 UTC
Created attachment 1195460 [details]
screenshot 20160829

screenshot goes with comment 69, 70, 71

Comment 73 David Lehman 2016-08-29 18:56:11 UTC
It looks like there are two things going on here:

1. the live image is not auto-activating md as all other system variants do
2. blivet has lost the ability to activate them since they're auto-activated everywhere except, apparently, on fedora live images

Someone needs to decide if fedora live images are not activating md by design or in error. It's probably a bug that blivet doesn't auto-activate them as needed, but I'm not particularly moved by arbitrary variations in the various installation environments.

Comment 74 Adam Williamson 2016-08-29 19:10:05 UTC
IIRC it's intentional that we don't automatically bring up any local storage when booting live, because we want to avoid any possibility of unexpected changes to it. But I'm not 100% sure. It would probably be a good idea to boot some old lives and see if they were auto-activating RAID sets or not.

Comment 75 David Lehman 2016-08-29 19:50:18 UTC
(In reply to Adam Williamson from comment #74)
> IIRC it's intentional that we don't automatically bring up any local storage
> when booting live, because we want to avoid any possibility of unexpected
> changes to it. But I'm not 100% sure. It would probably be a good idea to
> boot some old lives and see if they were auto-activating RAID sets or not.

If anything, I think having the storage active by default is likely to _prevent_ unintentional changes to it (think EBUSY). Either way, it would be good to communicate such a change to the installer team.

Comment 76 Adam Williamson 2016-08-29 19:56:26 UTC
I wasn't actually saying it was a change...

thinking a bit more, though, it may have changed a few years back when dracut changed its behaviour regarding auto-activating RAID sets...

Comment 77 Chris Murphy 2016-08-29 20:20:22 UTC
It's active before I launch the installer:
[liveuser@localhost ~]$ su
[root@localhost liveuser]# cat /proc/mdstat 
Personalities : [raid0] 
md127 : active raid0 vdb1[1] vda1[0]
      18890752 blocks super 1.2 512k chunks
      
unused devices: <none>

From the program.log:
14:16:14,827 INFO program: Running [14] mdadm --stop /dev/md/root ...

Comment 78 David Lehman 2016-08-30 15:12:55 UTC
Thanks for pointing that out -- I overlooked 'root' in the initial device list. You ran anaconda at least twice in that live session. What happened the first time?

Comment 79 David Lehman 2016-08-30 15:14:21 UTC
Hmm, or is all that previous output from anaconda-cleanup?

Comment 80 David Lehman 2016-08-30 15:34:07 UTC
blivet does indeed deactivate /dev/md/root as part of anaconda-cleanup. This is not necessarily desirable _before_ the installation, but that's another matter. The reason that shouldn't matter is anaconda then does 'udevadm trigger --subsystem=block --action=change', which should activate the md array again. The reason this doesn't work out is probably related to a relic from the past: udevadm control --env=ANACONDA=1

Chris, can you please try the f25 installation again but, this time, comment out the following lines:

/usr/lib/udev/rules.d/64-md-raid-assembly.rules:ENV{ANACONDA}=="?*", GOTO="md_inc_end"
/usr/lib/udev/rules.d/65-md-incremental.rules:ENV{ANACONDA}=="?*", GOTO="md_end"

(Just put a '#' at the start of each line.)

Then try the installation and see if things improve. Thanks.

Comment 81 Chris Murphy 2016-08-30 16:33:54 UTC
(In reply to David Lehman from comment #78)
> Thanks for pointing that out -- I overlooked 'root' in the initial device
> list. You ran anaconda at least twice in that live session. What happened
> the first time?

There was only one launch of anaconda in that live session. Is there any chance the udisks2>storaged change is involved in any confusion?


(In reply to David Lehman from comment #80)
> Chris, can you please try the f25 installation again but, this time, comment
> out the following lines:
> 
> /usr/lib/udev/rules.d/64-md-raid-assembly.rules:ENV{ANACONDA}=="?*",
> GOTO="md_inc_end"
> /usr/lib/udev/rules.d/65-md-incremental.rules:ENV{ANACONDA}=="?*",
> GOTO="md_end"

So boot the live media, then comment out these lines, then launch the installer?

Comment 82 David Lehman 2016-08-30 17:14:25 UTC
(In reply to Chris Murphy from comment #81)
> There was only one launch of anaconda in that live session. Is there any
> chance the udisks2>storaged change is involved in any confusion?

No, I just forgot all about anaconda-cleanup.

> So boot the live media, then comment out these lines, then launch the
> installer?

Correct.

Comment 83 Chris Murphy 2016-08-30 17:21:01 UTC
(In reply to David Lehman from comment #80)

> /usr/lib/udev/rules.d/64-md-raid-assembly.rules:ENV{ANACONDA}=="?*",
> GOTO="md_inc_end"
> /usr/lib/udev/rules.d/65-md-incremental.rules:ENV{ANACONDA}=="?*",
> GOTO="md_end"

No change.

Comment 84 Chris Murphy 2016-08-30 17:23:37 UTC
Created attachment 1196013 [details]
program.log 20160830

program.log from comment 83

Comment 85 Chris Murphy 2016-08-30 17:23:56 UTC
Created attachment 1196014 [details]
storage.log 20160830

storage.log from comment 83

Comment 86 David Lehman 2016-08-30 17:49:41 UTC
If you are still willing, please try once more, but this time run the following command after changing the rules and before starting anaconda:

  udevadm control --reload-rules


Thanks again.

Comment 87 Chris Murphy 2016-08-30 18:35:22 UTC
(In reply to David Lehman from comment #86)
> If you are still willing, please try once more, but this time run the
> following command after changing the rules and before starting anaconda:
> 
>   udevadm control --reload-rules

Yeah I always forget that...

OK so now we have a different outcome, but it's still something of an odd duck: It knows it's 9GiB, instead of 0 bytes. But it's still Unknown, and also even Unknown file system, so if we were to pretend this were a home array, I can't assign it the /home mount point. Unless it's saying it's unknown but really knows it's rootfs and hence not reusable?

Comment 88 Chris Murphy 2016-08-30 18:36:35 UTC
Created attachment 1196053 [details]
screenshot for comment 87

Comment 89 Chris Murphy 2016-08-30 18:38:06 UTC
Created attachment 1196054 [details]
storage.log c87

Comment 90 Chris Murphy 2016-08-30 18:38:19 UTC
Created attachment 1196055 [details]
program.log c87

Comment 91 Geoffrey Marr 2016-08-30 19:21:10 UTC
Discussed during the 2016-08-29 blocker review meeting: [1]

The decision to classify this bug as an AcceptedBlocker (Beta) was made as this bug is reproducible with clean disk images and violates the following Beta criteria:

"When using the custom partitioning flow, the installer must be able to correctly interpret...any disk with a valid ms-dos or gpt disk label and partition table containing ext4 partitions, LVM and/or btrfs volumes, and/or software RAID arrays at RAID levels 0, 1 and 5 containing ext4 partitions" for live images.

[1] https://meetbot.fedoraproject.org/fedora-blocker-review/2016-08-29/f25-blocker-review.2016-08-29-16.00.txt

Comment 92 Paul Howarth 2016-09-10 19:19:46 UTC
Just tried updating my son's LVM-on-mdraid1 box to Fedora 24, using Fedora 24 Server on a USB Stick as I've gotten used to the Workstation installer not working for this case, and this time the Server installer isn't working either. The LVM PV partitions just show up under "unknown".

I can run "vgscan", "vgchange -ay" and "vgdisplay" on another VT and all of the existing filesystems show up there but I can't get the installer to see them. Rescanning disks in the installer doesn't help. "rd.auto" on the kernel line doesn't help either. Is there any workaround I could try here or am I just going to have to skip F24? Repartitioning and losing data is not an option for me.

Comment 93 Adam Pribyl 2016-09-10 20:26:32 UTC
This is fine as an excercise for this bug, but if you want to update, simply start fedora-upgrade from inside of the existing installation or use recommended https://fedoraproject.org/wiki/DNF_system_upgrade - this should avoid anaconda/blivet/whatever installer.

Comment 94 Adam Williamson 2016-09-11 04:36:11 UTC
In fact if you just run GNOME Software and go to the 'Updates' tab you should see a nice graphical 'Upgrade to Fedora 24' option.

Comment 95 Paul Howarth 2016-09-11 12:24:16 UTC
Would that work from an F22 install too? I generally do fresh installs into an alternate set of filesystems, so I can fall back to the previous release if necessary, or refer to config files from the previous release etc. So I current have an up to date F23 and its predecessor F22 install available for upgrade. Normally I'd overwrite the F22 install with the F24 one but I suppose I could upgrade that instead.

Is it not surprising that an F24 Server install can't see the volumes? Might the netinstall be any better?

Comment 96 Adam Williamson 2016-09-11 15:22:22 UTC
"Is it not surprising that an F24 Server install can't see the volumes?"

Sure, it's not expected. Without any logs, though, we can't tell what's going on at all (whether you're seeing the same case as cmurf, or something else).

Yes, upgrading via dnf-plugin-upgrade works fine from F22 (don't recall if we enabled graphical upgrades from F22 or not).

Comment 97 Anders Blomdell 2016-09-12 11:33:47 UTC
(In reply to Paul Howarth from comment #92)
> Just tried updating my son's LVM-on-mdraid1 box to Fedora 24, using Fedora
> 24 Server on a USB Stick as I've gotten used to the Workstation installer
> not working for this case, and this time the Server installer isn't working
> either. The LVM PV partitions just show up under "unknown".
> 
> I can run "vgscan", "vgchange -ay" and "vgdisplay" on another VT and all of
> the existing filesystems show up there but I can't get the installer to see
> them. Rescanning disks in the installer doesn't help. "rd.auto" on the
> kernel line doesn't help either. Is there any workaround I could try here or
> am I just going to have to skip F24? Repartitioning and losing data is not
> an option for me.

Most probably fixed by the attachment in:

https://bugzilla.redhat.com/show_bug.cgi?id=1361611#c8

Comment 98 Fawzy 2016-09-25 20:53:06 UTC
I have NEC EXPRESS 5800/R120B-2 with BIOS raid; I'm facing the same issue when I install F25 Alpha, F24,F23,F22.

I have also tried Centos 7 and 7.2 with the same issue.

Only F21 64bit and Centos 6.8 64bit can detect BIOS raid H.D. and can be installed on this legacy H.W.

I guess it's the same Centos bug reported on this link https://bugs.centos.org/view.php?id=9813

Comment 99 Fawzy 2016-09-25 20:57:56 UTC
Created attachment 1204630 [details]
lspci Output

Comment 100 Fawzy 2016-09-25 20:59:48 UTC
Comment on attachment 1204630 [details]
lspci Output

this is the output of "lspci -k" from Centos 6.8 running the below kernel version: Linux localhost.localdomain 2.6.32-642.3.1.el6.x86_64 #1 SMP Tue Jul 12 18:30:56 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Comment 101 Adam Williamson 2016-09-26 16:46:20 UTC
So, back around comments 66 through 90, I think we had at least zero'ed in on a specific case here: detection of existing software RAID sets in the live installer. I'm not sure Paul's or Fawzy's comments are about the same situation at all (Paul, at least, was clearly using a non-live installer image). To avoid confusion, can you two please file separate bugs if you're reporting different cases? Thanks.

Comment 102 Geoffrey Marr 2016-09-26 16:52:07 UTC
Hi David,

We looked through this bug at the 2016-09-26 Blocker-Review-Meeting and were not exactly sure on where this bug stands. Can you clarify as to what your understanding of this bug is and if you think any progress is being made?

Thanks,
Geoff M. - QA
IRC: coremodule

Comment 103 David Lehman 2016-09-26 19:01:36 UTC
There are a variety of things reported here, most notably:

 1. failure to detect fwraid that uses DDF metadata

    - DDF metadata is not supported by either dmraid or mdadm,
      making it impossible for the installer to support it

 2. failure to detect Intel imsm/isw fwraid in live media installations

    - read on


There are some obsolete bits related to md in anaconda that are probably contributing to this:

1. setting ANACONDA udev env variable to prevent md udev rules from auto-assembling md arrays
2. deactivating all non-critical storage before & after running anaconda in the live environment

Those are both anaconda changes. I'm going to proceed with those and then we'll see where it stands.

Comment 104 Adam Williamson 2016-09-26 20:10:15 UTC
OK. I just have one more question: cmurf's reproducer definitely reads like he actually used *software* RAID, not any kind of firmware RAID. kparal's comment (66) is ambiguous, I cannot tell from his comment whether he used firmware or software RAID. Do you think 2. could affect software RAID as well as Intel firmware RAID?

I've adjusted the summary to at least specify we're talking about the *live* installer here, not anything else. Other reporters: if what you're reporting is basically #1, I think the story here is that anaconda cannot do anything about it if the underlying tools don't support that metadata format, so basically what you need is a feature request for mdadm and/or dmraid. If what you're reporting is something else, please file a new bug (or find a closer duplicate for your case). Thanks!

Comment 105 Chris Murphy 2016-09-26 21:20:04 UTC
My testing is mdadm RAID as created by the Fedora 24 installer. 

Fedora 25 Live installer shows the arrays' names in custom partitioning, but shows them as 0 bytes in size, they can't be reused only deleted, and it doesn't see the Fedora 24 install.

Fedora 25 netinstall shows the arrays' names in custom partitioning, and their sizes, and sees the Fedora 24 installation, and I can reuse the arrays, i.e. I can assign the Fedora 24 home to /home.

Comment 106 Kamil Páral 2016-09-27 07:10:35 UTC
(In reply to Adam Williamson from comment #104)
> kparal's comment 66 is ambiguous, I cannot tell from his comment whether he used
> firmware or software RAID. 

Sorry. I used software RAID, performed in a VM.

Comment 107 David Lehman 2016-09-27 13:59:45 UTC
(In reply to Adam Williamson from comment #104)
> firmware or software RAID. Do you think 2. could affect software RAID as
> well as Intel firmware RAID?

Yes, I think it could.

Comment 108 David Lehman 2016-09-27 14:46:48 UTC
Upstream pull request:

  https://github.com/rhinstaller/anaconda/pull/812

Comment 109 Adam Williamson 2016-09-27 23:06:21 UTC
So, the fix merged by dlehman does fix the software RAID case for me. To test, I did a software RAID install to a VM with two clean VirtIO disks with Fedora-Workstation-Live-x86_64-25-20160927.n.0.iso . Then I booted back to the same image, ran the installer again, and verified it behaves as described by cmurf and kparal: the RAID set is detected but shows in custom partitioning as 0 bytes in size under 'Unknown'. Then I rebooted and updated to an anaconda scratch build with dlehman's patches - http://koji.fedoraproject.org/koji/taskinfo?taskID=15834767 - and ran the installer again: now the partition on the RAID set is detected with its correct size and appears under 'Fedora 25' in custom partitioning.

For the record, on my bare metal test system, current F25 nightlies cannot detect an Intel firmware RAID set *at all* with or without dlehman's patches. But this is clearly something different. I'll file a separate bug for that.

Comment 110 Fawzy 2016-10-03 19:11:18 UTC
I have tried also with latest F25 netinstall CD without dlehman's patches ; and it can not detect my Intel firmware RAID set.

I tried with F21 and upgraded it to F24 and F24 is working fine now on my Intel firmware RAID !!

Is there a hope to get F25 anaconda detects my Intel firmware raid and install on my legacy servers?

Comment 111 Adam Williamson 2016-10-03 19:18:15 UTC
The firmware RAID detection bug is https://bugzilla.redhat.com/show_bug.cgi?id=1379865 .

Comment 112 Adam Williamson 2016-10-03 22:39:19 UTC
Re-assigning to anaconda as that's where the fix went, setting MODIFIED as the PR has been merged, and setting appropriate fixed-in version.

Comment 113 Fedora Update System 2016-10-05 20:28:58 UTC
anaconda-25.20.4-1.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2016-067d26633e

Comment 114 Adam Williamson 2016-10-06 17:09:01 UTC
Verified the fix in Beta-1.1.

Comment 115 Fedora Update System 2016-10-07 03:34:21 UTC
anaconda-25.20.4-1.fc25 has been pushed to the Fedora 25 stable repository. If problems still persist, please make note of it in this bug report.

Comment 116 Ivan Talalaev 2017-05-01 20:46:19 UTC
Unfortunatelly the BUG stil persists,

I have faced it when created RAD0 array on ASROck N68C-s UCC matherboard, 

RAID0 array of type nvidia_ceeiebdh

I have tested it on Fedora 25 Workstation, and Fedora 25 Server edition.

a few lines from traceback 
...blivet/osinstall.py line 1175 in storage_initialize storate_reset()
...python3.5/threading.py line 862 in run self_target(*self_args, **self_kwargs)
...pyanaconda/threads.py line 251 in run threading.Thread.run(self, *args, **kwargs)
gi.overrides.BlockDev.DMError. Failed to activate the RAID set 'nvidia_ceeiebdh'

Comment 117 Adam Williamson 2017-05-02 00:36:58 UTC
That isn't at all the same bug as this. Please file a new bug.


Note You need to log in before you can comment on or make changes to this bug.