Bug 1145783 - F21 install crashes on Intel firmware RAID with "AttributeError: 'NoneType' object has no attribute 'startswith'"
Summary: F21 install crashes on Intel firmware RAID with "AttributeError: 'NoneType' o...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: python-blivet
Version: 21
Hardware: x86_64
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: mulhern
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: abrt_hash:772906dd7568eb88321b00538be...
: 1148074 (view as bug list)
Depends On:
Blocks: F21BetaBlocker 1150147
TreeView+ depends on / blocked
 
Reported: 2014-09-23 18:18 UTC by Joseph Bowman
Modified: 2017-10-08 04:25 UTC (History)
14 users (show)

Fixed In Version: python-blivet-0.61.4-1
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1150147 (view as bug list)
Environment:
Last Closed: 2014-10-15 21:28:42 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
File: anaconda-tb (825.26 KB, text/plain)
2014-09-23 18:18 UTC, Joseph Bowman
no flags Details
File: anaconda.log (39.33 KB, text/plain)
2014-09-23 18:18 UTC, Joseph Bowman
no flags Details
File: environ (616 bytes, text/plain)
2014-09-23 18:18 UTC, Joseph Bowman
no flags Details
File: journalctl (457.62 KB, text/plain)
2014-09-23 18:18 UTC, Joseph Bowman
no flags Details
File: lsblk_output (3.77 KB, text/plain)
2014-09-23 18:18 UTC, Joseph Bowman
no flags Details
File: nmcli_dev_list (2.28 KB, text/plain)
2014-09-23 18:18 UTC, Joseph Bowman
no flags Details
File: os_info (377 bytes, text/plain)
2014-09-23 18:18 UTC, Joseph Bowman
no flags Details
File: program.log (43.10 KB, text/plain)
2014-09-23 18:18 UTC, Joseph Bowman
no flags Details
File: storage.log (237.83 KB, text/plain)
2014-09-23 18:18 UTC, Joseph Bowman
no flags Details
File: ifcfg.log (6.77 KB, text/plain)
2014-09-23 18:18 UTC, Joseph Bowman
no flags Details
The contents of the anaconda error handler after applying the updates.img (742.42 KB, text/plain)
2014-10-02 02:09 UTC, Jeremy Rimpo
no flags Details
Anaconda error log 21 alpha 1 - applying updates.img #1 (1022.89 KB, text/plain)
2014-10-02 19:56 UTC, Jeremy Rimpo
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1141398 0 unspecified CLOSED anaconda does not see existing Fedora 21 install to LVM-on-RAID 2021-02-22 00:41:40 UTC
Red Hat Bugzilla 1147087 0 unspecified CLOSED Previous software raid disks not detected 2021-02-22 00:41:40 UTC
Red Hat Bugzilla 1153096 1 high CLOSED AttributeError: 'NoneType' object has no attribute 'startswith' 2024-04-25 07:26:46 UTC
Red Hat Bugzilla 1155288 0 unspecified CLOSED AttributeError: 'NoneType' object has no attribute 'startswith' 2021-02-22 00:41:40 UTC

Internal Links: 1141398 1147087 1153096 1155288

Description Joseph Bowman 2014-09-23 18:18:14 UTC
Description of problem:
Started "install to hard drive", clicked next and after a second it crashed. I am in liveboot.

Version-Release number of selected component:
anaconda-core-21.48.6-1.fc21.x86_64

The following was filed automatically by anaconda:
anaconda 21.48.6-1 exception report
Traceback (most recent call first):
  File "/usr/lib/python2.7/site-packages/blivet/devices.py", line 1148, in isNameValid
    if name.startswith("cciss/"):
  File "/usr/lib/python2.7/site-packages/blivet/devices.py", line 244, in __init__
    if not self.isNameValid(name):
  File "/usr/lib/python2.7/site-packages/blivet/devices.py", line 555, in __init__
    Device.__init__(self, name, parents=parents)
  File "/usr/lib/python2.7/site-packages/blivet/devices.py", line 2270, in __init__
    super(ContainerDevice, self).__init__(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/blivet/devices.py", line 3593, in __init__
    sysfsPath=sysfsPath)
  File "/usr/lib/python2.7/site-packages/blivet/devicetree.py", line 1642, in handleUdevMDMemberFormat
    exists=True)
  File "/usr/lib/python2.7/site-packages/blivet/devicetree.py", line 1889, in handleUdevDeviceFormat
    self.handleUdevMDMemberFormat(info, device)
  File "/usr/lib/python2.7/site-packages/blivet/devicetree.py", line 1244, in addUdevDevice
    self.handleUdevDeviceFormat(info, device)
  File "/usr/lib/python2.7/site-packages/blivet/devicetree.py", line 2170, in _populate
    self.addUdevDevice(dev)
  File "/usr/lib/python2.7/site-packages/blivet/devicetree.py", line 2105, in populate
    self._populate()
  File "/usr/lib/python2.7/site-packages/blivet/__init__.py", line 469, in reset
    self.devicetree.populate(cleanupOnly=cleanupOnly)
  File "/usr/lib/python2.7/site-packages/blivet/__init__.py", line 188, in storageInitialize
    storage.reset()
  File "/usr/lib64/python2.7/threading.py", line 766, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib64/python2.7/site-packages/pyanaconda/threads.py", line 227, in run
    threading.Thread.run(self, *args, **kwargs)
AttributeError: 'NoneType' object has no attribute 'startswith'

Additional info:
cmdline:        /usr/bin/python  /sbin/anaconda --liveinst --method=livecd:///dev/mapper/live-base
cmdline_file:   BOOT_IMAGE=/syslinux/vmlinuz0 root=live:LABEL=LIVE ro rd.live.image quiet rhgb
executable:     /sbin/anaconda
hashmarkername: anaconda
kernel:         3.16.1-301.fc21.x86_64
other involved packages: python-blivet-0.62-1.fc21.noarch, python-libs-2.7.8-4.1.fc21.x86_64
product:        Fedora
release:        Fedora release 21 (Twenty One)
type:           anaconda
version:        21

Comment 1 Joseph Bowman 2014-09-23 18:18:16 UTC
Created attachment 940538 [details]
File: anaconda-tb

Comment 2 Joseph Bowman 2014-09-23 18:18:17 UTC
Created attachment 940539 [details]
File: anaconda.log

Comment 3 Joseph Bowman 2014-09-23 18:18:18 UTC
Created attachment 940540 [details]
File: environ

Comment 4 Joseph Bowman 2014-09-23 18:18:19 UTC
Created attachment 940541 [details]
File: journalctl

Comment 5 Joseph Bowman 2014-09-23 18:18:20 UTC
Created attachment 940542 [details]
File: lsblk_output

Comment 6 Joseph Bowman 2014-09-23 18:18:21 UTC
Created attachment 940543 [details]
File: nmcli_dev_list

Comment 7 Joseph Bowman 2014-09-23 18:18:22 UTC
Created attachment 940544 [details]
File: os_info

Comment 8 Joseph Bowman 2014-09-23 18:18:23 UTC
Created attachment 940545 [details]
File: program.log

Comment 9 Joseph Bowman 2014-09-23 18:18:24 UTC
Created attachment 940546 [details]
File: storage.log

Comment 10 Joseph Bowman 2014-09-23 18:18:25 UTC
Created attachment 940547 [details]
File: ifcfg.log

Comment 11 Joseph Bowman 2014-09-23 20:56:53 UTC
Also, I have used fedora for a few years however this is my first bug report so please advice if there is any more information that I can provide to assist. In case the logs do not easily provide this:

This is on an Asus Rog G750-JH Laptop, This laptop has 2 PCI-E SSD's in a raid 0 with a sata hdd as a secondary drive. It seams to crash as it is "probing" installation media. 

After submitting this repot I have removed the secondary sata drive to test if that was the cause and the issue still persisted so I am guessing, without being a programer, that this may be related to the raid config.

Comment 12 Brian Lane 2014-10-01 00:27:32 UTC
*** Bug 1148074 has been marked as a duplicate of this bug. ***

Comment 13 Vratislav Podzimek 2014-10-01 08:17:10 UTC
From the anaconda-tb file:
> Local variables in innermost frame:
> name: None
> cls: <class 'blivet.devices.MDRaidArrayDevice'>

So we have an MDRaidArrayDevice with no name (given by udev probably). mulhern, any idea what could be going on here? Could it be another instance of the "UUID format" issue?

Comment 14 Jeremy Rimpo 2014-10-01 15:07:33 UTC
In my case, I have a standalone SSD (unused by this install) and an MD SATA raid.

There is definitely no issue on the F20 anaconda.

Comment 15 mulhern 2014-10-01 15:31:50 UTC
I can not say. There is no telltale poorly formatted UUID anywhere, unlike in bz#1147087. However, absence of evidence is not evidence of absence. dlehman might know more...there is a lot of name finding that occurs in handleUdevMDMemberFormat() that he worked w/ recently.

Comment 16 David Lehman 2014-10-01 17:07:30 UTC
I'd say there is a good chance that this is caused by UUID formatting.

Comment 17 mulhern 2014-10-01 21:25:45 UTC
We suspect that this problem may be due to an inconsistency between the way mdadm formats UUIDs and the way the rest of the world does.

I've posted an updates.img at:
http://mulhern.fedorapeople.org/1145783.img. It contains a fix for the bug (we think),
and some extra logging code as well.

Can you run the install again with this image applied, let us know how it goes, and also upload logs for that run as well?

Please let me know if you have any questions.

Thanks!

- mulhern

Comment 18 Jeremy Rimpo 2014-10-02 00:36:54 UTC
It seems like I'm getting the same error. All I have to do is place the updates.img in the /tmp right? (Also tried with boot: linux0 updates=http://...) Which log files do I need to post?

Comment 19 Jeremy Rimpo 2014-10-02 02:09:16 UTC
Created attachment 943254 [details]
The contents of the anaconda error handler after applying the updates.img

Here is the output of the error handler when anaconda crashes.

Comment 20 mulhern 2014-10-02 11:26:15 UTC
Hi Jeremy,

The updates.img was not applied...I can tell by lack of additional error output in anaconda-tb. I always use inst.updates at boot.

Comment 21 mulhern 2014-10-02 11:28:12 UTC
Re: Comment #19.

Comment 22 Joseph Bowman 2014-10-02 12:53:06 UTC
Good Morning all,

I will run this when I get home from work today. 

Thank you,
Joe

Comment 23 Jeremy Rimpo 2014-10-02 15:28:38 UTC
Does this need to be applied to a specific iso? But I may be having different problems. Right now I can't seem to even get the text installer to run - it's always launching the default live desktop no matter what options I select - even through the boot menu.

Comment 24 Jeremy Rimpo 2014-10-02 15:47:28 UTC
I got a different result - sort of. I now appear to get two ABRT reports, seconds apart, and the backtrace of the second one lists:

base.py:159:_thread_input:EOFError: EOF when reading a line

Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/pyanaconda/threads.py", line 227, in run
    threading.Thread.run(self, *args, **kwargs)
  File "/usr/lib64/python2.7/threading.py", line 766, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib64/python2.7/site-packages/pyanaconda/ui/tui/simpleline/base.py", line 159, in _thread_input
    data = raw_input()
EOFError: EOF when reading a line

Local variables in innermost frame:
queue: <Queue.Queue instance at 0x7f3e936c91b8>
hidden: False
prompt: u"  Please make your choice from above ['q' to quit | 'c' to continue |\n  'r' to refresh]: "
self: <pyanaconda.ui.tui.simpleline.base.App object at 0x7f3e936b7b10>

Comment 25 Jeremy Rimpo 2014-10-02 16:10:04 UTC
I think the updates.img is being applied now. I'm getting the same result both using a URL to the above file as well as off a second USB. Unfortunately, I no longer even get to the anaconda error report, and the anaconda.log in /tmp looks equally unhelpful. It mentions the logging level is debug. It starts AnaInputThread1 which then immediately stops and the log ends. As for the ABRT reports, I get one that looks like the original error followed quickly by one with the above backtrace. Is there another file I should look for?

I should mention I'm using the recent Beta TC1 iso. I'm not sure if that would make a difference.

Comment 26 Jeremy Rimpo 2014-10-02 16:44:19 UTC
I should mention, the updates.img I see downloaded to the /tmp directory when using a URL in the inst.updates= is smaller than the one I download myself. I would guess it's not able to process the whole img for some reason, but I don't know why. I'm going to try again on the official alpha release just to see what happens.

Comment 27 David Lehman 2014-10-02 17:11:24 UTC
I don't think we support running a text-mode install from live media. I can't think of a reason to do such a thing.

Comment 28 Brian Lane 2014-10-02 17:12:10 UTC
Note that for repeated runs it may be easier to just run liveinst from a terminal, like this:

liveinst --updates=http://path/to/updates.img

Comment 29 Jeremy Rimpo 2014-10-02 19:56:36 UTC
Created attachment 943543 [details]
Anaconda error log 21 alpha 1 - applying updates.img #1

After some trial and error, I got the img to apply and made some progress. Whereas before anaconda crashed before even selecting language, I can now make it slightly past that point before a new error is triggered.

Attaching the contents of the anaconda error report.

Comment 30 Vratislav Podzimek 2014-10-02 20:12:18 UTC
(In reply to Jeremy Rimpo from comment #29)
> Created attachment 943543 [details]
> Anaconda error log 21 alpha 1 - applying updates.img #1
> 
> After some trial and error, I got the img to apply and made some progress.
> Whereas before anaconda crashed before even selecting language, I can now
> make it slightly past that point before a new error is triggered.
> 
> Attaching the contents of the anaconda error report.
Interesting::
Traceback (most recent call first):
  File "/tmp/updates/blivet/devices.py", line 2347, in _addParent
    raise ValueError("member has wrong format")
...


Local variables in innermost frame:
member: existing 1863.03 GiB mdcontainer imsm0 (20)

Comment 31 Jeremy Rimpo 2014-10-06 14:50:51 UTC
This may or may not be related to the newer error, but it may help debug in the off chance. My RAID drive is an Intel Software Raid of two 1TB drives (as you most likely already noticed) which is further partitioned into one large data portion (I think 1.6 TB) as NTFS and the rest is used by Fedora as a mix of Ext4 as root and and LVM2 as home data. Using MBR partitioning.

Comment 32 Jeremy Rimpo 2014-10-06 15:12:21 UTC
Correction, there's a Ext4 boot partition, and the LVM is a mix of root, home, and swap partitions.

Comment 33 Adam Williamson 2014-10-08 21:57:35 UTC
Proposing as a 21 Beta blocker, criterion "The installer must be able to detect and install to hardware or firmware RAID storage devices." - I've reproduced both bugs here with Intel firmware RAID. TC2 (with blivet 0.61.3) hits the first crash (has no attribute 'startswith'). If I apply an updates.img with the changes in blivet 0.61.4 it hits the "member has wrong format" crash. This is all from the Server DVD, no liveinst / text mode shenanigans.

Comment 34 Adam Williamson 2014-10-08 22:00:53 UTC
Looks like Anne wanted to split the second crash out as #1150147. If so, this one should be marked as ON_QA because this specific crash (the 'startswith' one) is fixed in blivet 0.61.4 .

Comment 35 mulhern 2014-10-09 11:17:06 UTC
(In reply to Adam Williamson (Red Hat) from comment #34)
> Looks like Anne wanted to split the second crash out as #1150147. If so,
> this one should be marked as ON_QA because this specific crash (the
> 'startswith' one) is fixed in blivet 0.61.4 .

Agreed, I'm marking as modified, because AFAIK the originally reported bug is fixed.

Comment 36 Adam Williamson 2014-10-15 16:19:09 UTC
Discussed at 2014-10-15 blocker review meeting: http://meetbot.fedoraproject.org/fedora-blocker-review/2014-10-15/f21-blocker-review.2014-10-15-16.04.log.txt . Accepted as a blocker per criterion cited in c#33.

Comment 37 Adam Williamson 2014-10-15 21:28:42 UTC
0.61.4 is now pending stable and this is verified fixed in it, so let's just close it out and keep the blocker list clean. We have #1150147 to work the second crash (should be fixed in 0.61.5 / TC4).

Comment 38 Ben Galliart 2017-10-06 03:19:19 UTC
I recreated this bug with CentOS 7.4 (Anaconda 21.48.22.121-1, Blivet 0.61.15.65) with a backtrace of:

  File "/usr/lib64/python2.7/site-packages/pyanaconda/threads.py", line 227, in 
run
    threading.Thread.run(self, *args, **kwargs)

  File "/usr/lib64/python2.7/threading.py", line 765, in run
    self.__target(*self.__args, **self.__kwargs)

  File "/usr/lib/python2.7/site-packages/blivet/__init__.py", line 190, in storageInitialize
    storage.reset()

  File "/usr/lib/python2.7/site-packages/blivet/__init__.py", line 495, in reset
    self.devicetree.populate(cleanupOnly=cleanupOnly)

  File "/usr/lib/python2.7/site-packages/blivet/devicetree.py", line 2239, in populate
    self._populate()

  File "/usr/lib/python2.7/site-packages/blivet/devicetree.py", line 2306, in _populate
    self.addUdevDevice(dev)

  File "/usr/lib/python2.7/site-packages/blivet/devicetree.py", line 1277, in addUdevDevice
    self.handleUdevDeviceFormat(info, device)

  File "/usr/lib/python2.7/site-packages/blivet/devicetree.py", line 1988, in handleUdevDeviceFormat
    self.handleUdevMDMemberFormat(info, device)

  File "/usr/lib/python2.7/site-packages/blivet/devicetree.py", line 1729, in handleUdevMDMemberFormat
    exists=True)

  File "/usr/lib/python2.7/site-packages/blivet/devices/md.py", line 106, in __init__
    sysfsPath=sysfsPath)

  File "/usr/lib/python2.7/site-packages/blivet/devices/container.py", line 61, in __init__
    super(ContainerDevice, self).__init__(*args, **kwargs)

  File "/usr/lib/python2.7/site-packages/blivet/devices/storage.py", line 146, in __init__
    Device.__init__(self, name, parents=parents)

  File "/usr/lib/python2.7/site-packages/blivet/devices/device.py", line 83, in __init__
    if not self.isNameValid(name):

  File "/usr/lib/python2.7/site-packages/blivet/devices/storage.py", line 854, in isNameValid
    if name.startswith("cciss/"):

AttributeError: 'NoneType' object has no attribute 'startswith'

In StorageDevice.__init__ right before Device.__init__ is called I added 
print("name: " + str(name))
print("uuid: " + str(uuid))

The result I got right before the AttributeError was raised as:
name: None
uuid: 462b445e-a58a-2956-10fd-3b49b91e9582

I then located that UUID using 'mdadm -E /dev/sdb' which shows:
/dev/sdb:
          Magic : Intel Raid ISM Cfg Sig.
        Version : 1.1.00
    Orig Family : ec19e999
         Family : ec19e999
     Generation : 001f8f5a
     Attributes : All supported
           UUID : 462b445e:a58a2956:10fd3b49:b91e9582
       Checksum : 59648136 correct
    MPB Sectors : 1
          Disks : 2
   RAID Devices : 1

  Disk01 Serial : 6VMQHQ9X
          State : active
             Id : 00000001
    Usable Size : 976767240 (465.76 GiB 500.10 GB)

[Primary Mirror]:
           UUID : df2b8b10:cf3885e4:0474f472:0e6a4a22
     RAID Level : 1
        Members : 2
          Slots : [UU]
    Failed disk : none
      This Slot : 1
    Sector Size : 512
     Array Size : 976766976 (465.76 GiB 500.10 GB)
   Per Dev Size : 976767240 (465.76 GiB 500.10 GB)
  Sector Offset : 0
    Num Stripes : 3815496
     Chunk Size : 64 KiB
       Reserved : 0
  Migrate State : idle
      Map State : normal
    Dirty State : clean
     RWH Policy : off

  Disk00 Serial : 6VM6XGBA
          State : active
             Id : 00000000
    Usable Size : 976767240 (465.76 GiB 500.10 GB)

As a work around for this bug, I ended up running:
mdadm --zero-superblock /dev/sdb

Comment 39 Ben Galliart 2017-10-06 06:14:09 UTC
As a side note, when I recreated the bug, the Anaconda kickstart included the following:

zerombr
clearpart --all --initlabel --drives=sda,sdb

You may want to consider extending the Anaconda clearpart support to attempt doing a mdadm --zero-superblock against the specified drives to reduce the chance of this issue in the future.

Comment 40 Jiri Konecny 2017-10-06 08:11:41 UTC
Hello,

This issue is fixed for a long time. If you are experiencing it now please file a new bug and attach your logs to the new bug.

Thank you,
Jirka

Comment 41 Ben Galliart 2017-10-07 01:59:31 UTC
Going carefully through the history of the bug, it seems like there is a race condition with the superclass constructor initialization ordering which can pass through uninitialized values (such as passing a type of None for the name instead of a string).  The verified fix seems to be nothing more than get the user to re-run Anaconda enough times until the race condition isn't triggered.  There is also no additional exception handling around Device.__init__ to try to provide meaningful data back for when the problem happens again and confirm if the bug was really related to uuid formatting.

If the same exact event of Device.__init__() getting a name of None and passing it to isNameValid() just to generate an uncaught exception when attempting to treat None as a string is considered to be "fixed," then I will just consider this crash to be the intended result.  There is no point in opening a new bug for something already considered fixed.  I already did my part in notifying Red Hat that the issue still exists.

Comment 42 Ben Galliart 2017-10-08 04:25:35 UTC
I'm sorry, I didn't take into account that RHEL 7 is based off Fedora 19.  You are right this is an old bug which has been fixed.  The issue can't be recreated in Fedora 26.

I will only open a new bug if I can generate an issue with RHEL 8beta (whenever that becomes available).

For anyone running across this ticket because of issues with RHEL 7 when use of Intel Raid isn't intended, I recommend again the workaround of issues the following against each drive:

mdadm --zero-superblock /dev/sda

Thanks


Note You need to log in before you can comment on or make changes to this bug.