Bug 1691016

Summary: anaconda failing with core in python3 for f30 or Rawhide reported in openQA tests ppc64le
Product: [Fedora] Fedora Reporter: Michel Normand <normand>
Component: anacondaAssignee: Anaconda Maintenance Team <anaconda-maint-list>
Status: CLOSED DUPLICATE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: high    
Version: 30CC: anaconda-maint-list, awilliam, dan, jkonecny, jonathan, kellin, vanmeeuwen+fedora, vponcova, wwoods
Target Milestone: ---   
Target Release: ---   
Hardware: powerpc   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-03-29 10:28:34 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1071880    
Attachments:
Description Flags
_boot_to_anaconda-journalctl.log.gz none

Description Michel Normand 2019-03-20 16:21:22 UTC 
anaconda failing with core in python3 for f30 or Rawhide reported in openQA tests

problem identified with 
* Rawhide compose 20190320 as per url (1)

* f30 compose 20190319 locally tested in IBM openQA

I retrieved the core file from uploaded data (2) and identified call trace (but do not have debug source files)
===
$gdb -c /tmp/tmp/anaconda.core.2088
GNU gdb (GDB) Fedora 8.2-6.fc29
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "ppc64le-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".   
[New LWP 2088]
[New LWP 2179]
[New LWP 2180]
[New LWP 2381]
[New LWP 2382]
[New LWP 2393]
[New LWP 2443]
Core was generated by `/usr/bin/python3 '.
#0  0x00007fffb530d864 in ?? ()
[Current thread is 1 (LWP 2088)]
(gdb) bt
#0  0x00007fffb530d864 in ?? ()
#1  0x00007fffb530d83c in ?? ()
#2  0x00007fffa4101404 in ?? ()
#3  <signal handler called>
#4  0x00007fff9e2448c8 in ?? ()
#5  0x00007fff9e317648 in ?? ()
#6  0x00007fff9e317550 in ?? ()
#7  0x00007fff9e311fe4 in ?? ()
#8  0x00007fff9e312460 in ?? ()
#9  0x00007fff9e3124fc in ?? ()
#10 0x00007fff9e3a19a4 in ?? ()
#11 0x00007fff9e398b14 in ?? ()
#12 0x00007fff9eab4a6c in ?? ()
#13 0x00007fff9eacf758 in ?? ()
#14 0x00007fff9eaba328 in ?? ()
#15 0x00007fff9eabcb4c in ?? ()
#16 0x00007fff9eac0260 in ?? ()
#17 0x00007fff9eac2678 in ?? ()
#18 0x00007fff9d823110 in ?? ()
#19 0x00007fff9d817a30 in ?? ()
#20 0x00007fff9d80a074 in ?? ()
#21 0x00007fff9d80b288 in ?? ()
#22 0x00007fff9d80d904 in ?? ()
#23 0x00007fff9d804700 in ?? ()
#24 0x00007fff9db7471c in ?? ()
#25 0x00007fff9db519f4 in ?? ()
#26 0x00007fff9db5ca28 in ?? ()
#27 0x00007fff9db5d3fc in ?? ()
#28 0x00007fff9da838a8 in ?? ()
#29 0x00007fff9da84180 in ?? ()
#30 0x00007fff9da62124 in ?? ()
#31 0x00007fff9d85d9cc in ?? ()
#32 0x00007fff9d864454 in ?? ()
#33 0x00007fff9da5cc98 in ?? ()
#34 0x00007fff9da838a8 in ?? ()
#35 0x00007fff9da84180 in ?? ()
#36 0x00007fff9d932564 in ?? ()
#37 0x00007fff9d93288c in ?? ()
#38 0x00007fff9d85d9cc in ?? ()
#39 0x00007fff9d864454 in ?? ()
#40 0x00007fff9d92f568 in ?? ()
#41 0x00007fff9da838a8 in ?? ()
#42 0x00007fff9da84180 in ?? ()
#43 0x00007fff9d7dc81c in ?? ()
#44 0x00007fff9d85d9cc in ?? ()
#45 0x00007fff9d864454 in ?? ()
#46 0x00007fff9d7dd808 in ?? ()
#47 0x00007fff9da838a8 in ?? ()
#48 0x00007fff9da84180 in ?? ()
#49 0x00007fff9d729fd0 in ?? ()
#50 0x00007fff9da838a8 in ?? ()
#51 0x00007fff9da84180 in ?? ()
#52 0x00007fff9d7dc81c in ?? ()
#53 0x00007fff9d85d9cc in ?? ()
#54 0x00007fff9d864454 in ?? ()
#55 0x00007fff9d7dd808 in ?? ()
#56 0x00007fff9da838a8 in ?? ()
#57 0x00007fff9da84180 in ?? ()
#58 0x00007fff9d7d5590 in ?? ()
#59 0x00007fff9da838a8 in ?? ()
#60 0x00007fff9da84180 in ?? ()
#61 0x00007fff9da8e34c in ?? ()
#62 0x00007fff9d85d9cc in ?? ()
#63 0x00007fff9d864454 in ?? ()
#64 0x00007fff9da8f6f0 in ?? ()
#65 0x00007fff9da838a8 in ?? ()
#66 0x00007fff9da84180 in ?? ()
#67 0x00007fff9d7d5590 in ?? ()
#68 0x00007fff9da838a8 in ?? ()
#69 0x00007fff9da84180 in ?? ()
#70 0x00007fff9dba78b0 in ?? ()
#71 0x00007fff9da838a8 in ?? ()
#72 0x00007fff9da84180 in ?? ()
#73 0x00007fff9da84a20 in ?? ()
#74 0x00007fff9dba98ac in ?? ()
#75 0x00007fff9dbb3f28 in ?? ()
#76 0x00007fffa6e0ac4c in ?? ()
#77 0x00007fffa6e04e78 in ?? ()
#78 0x00007fffa6e07a20 in ?? ()
#79 0x00007fffa6e2fc54 in ?? ()
#80 0x00007fffa6e30240 in ?? ()
#81 0x00007fff9d8526ec in ?? ()
#82 0x00007fff9d8529b8 in ?? ()
#83 0x00007fffa6e0abb0 in ?? ()
#84 0x00007fffa6e0768c in ?? ()
#85 0x00007fffa6e257e8 in ?? ()
#86 0x00007fffa6e2fe5c in ?? ()
#87 0x00007fffa6e30240 in ?? ()
#88 0x00007fff9e883c44 in ?? ()
#89 0x00007fff9e88498c in ?? ()
#90 0x00007fff9e861680 in ?? ()
#91 0x00007fffa6f6be18 in ?? ()
#92 0x00007fffa6f6ac2c in ?? ()
#93 0x00007fffa6f6b148 in ?? ()
#94 0x00007fffa6f6b73c in ?? ()
#95 0x00007fff9d998b34 in ?? ()
#96 0x00007fffa6dc74e0 in ?? ()
#97 0x00007fffa6dc6084 in ?? ()
#98 0x00007fffa70d4f28 in ?? ()
#99 0x00007fffa70d5f28 in ?? ()
#100 0x00007fffa70d7b58 in ?? ()
#101 0x00007fffa70d5b34 in ?? ()
#102 0x00007fffa70bea04 in ?? ()
#103 0x00007fffb4f0b530 in ?? ()
#104 0x00007fffb4e97248 in ?? ()
#105 0x00007fffb4ed03a4 in ?? ()
#106 0x00007fffb4f9c880 in ?? ()
#107 0x00007fffb4f9cb68 in ?? ()
#108 0x00007fffb4e916f4 in ?? ()
#109 0x00007fffb4e99990 in ?? ()
#110 0x00007fffb4ed03a4 in ?? ()
#111 0x00007fffb4e91444 in ?? ()
#112 0x00007fffb4e916f4 in ?? ()
#113 0x00007fffb4e92fd4 in ?? ()
#114 0x00007fffb4ed03a4 in ?? ()
#115 0x00007fffb4f9c880 in ?? ()
#116 0x00007fffb4f9ca14 in ?? ()
#117 0x00007fffb4f9ca8c in ?? ()
#118 0x00007fffb50824e8 in ?? ()
#119 0x00007fffb5083560 in ?? ()
#120 0x00007fffb50ae624 in ?? ()
#121 0x00007fffb50aeddc in ?? ()
#122 0x00007fffb50b4028 in ?? ()
#123 0x00007fffb50b4530 in ?? ()
#124 0x0000000105870838 in ?? ()
#125 0x00007fffb5248728 in ?? ()
#126 0x00007fffb5248924 in ?? ()
#127 0x0000000000000000 in ?? ()
===

(1) https://openqa.stg.fedoraproject.org/tests/499908#step/_boot_to_anaconda/7
(2) https://openqa.stg.fedoraproject.org/tests/499908/file/_boot_to_anaconda-anaconda.core.tar.gz
(3) https://openqa.stg.fedoraproject.org/tests/499908/file/_boot_to_anaconda-anaconda.log
Comment 1 Dan Horák 2019-03-20 16:56:02 UTC 
I think it's worth to mention that the failure/crash happened only in very few test cases, the majority went successfully (at least in Fedora OpenQA).
Comment 2 Jiri Konecny 2019-03-21 08:50:56 UTC 
Hello,

Thanks for your report. Could you please provide us journal log and other anaconda logs. Also it would be great if you can reproduce this with debug data.
Comment 3 Michel Normand 2019-03-21 09:57:27 UTC 
Hi Jiri,
what are the needed anaconda logs that are missing in what is already available ?
https://openqa.stg.fedoraproject.org/tests/499908#downloads
    _boot_to_anaconda-anaconda.core.tar.gz
    _boot_to_anaconda-anaconda.log
    _boot_to_anaconda-dbus.log
    _boot_to_anaconda-dnf.librepo.log
    _boot_to_anaconda-packaging.log
    _boot_to_anaconda-program.log
    _boot_to_anaconda-storage.log
    _boot_to_anaconda-syslog
    _boot_to_anaconda-var_log.tar.gz
    _boot_to_anaconda-var_tmp.tar.gz
    _boot_to_anaconda-X.log

For the journal log I will do a retry to capture it.
What are the debug data, and how to obtain it ? is it different from the inst.loglevel=debug boot option ?
Comment 4 Michel Normand 2019-03-22 08:27:00 UTC 
Created attachment 1546799 [details]
_boot_to_anaconda-journalctl.log.gz

journalctl captured from a new trial, uploaded here, extracted lines:

===

Mar 21 20:06:04 localhost org.fedoraproject.Anaconda.Modules.Storage[2271]: WARNING:py.warnings:/usr/lib64/python3.7/site-packages/pyanaconda/core/util.py:36: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
Mar 21 20:06:04 localhost org.fedoraproject.Anaconda.Modules.Storage[2271]:   import imp
Mar 21 20:06:05 localhost anaconda[2187]: anaconda: misc: Waiting 594 sec for modules to be started.
Mar 21 20:06:05 localhost org.fedoraproject.Anaconda.Modules.Storage[2271]: DEBUG:blivet:IGNORED:                             Caught exception, continuing.
Mar 21 20:06:05 localhost org.fedoraproject.Anaconda.Modules.Storage[2271]: DEBUG:blivet:IGNORED:                             Problem description: failed to get initiator name from iscsi firmware
Mar 21 20:06:05 localhost org.fedoraproject.Anaconda.Modules.Storage[2271]: DEBUG:blivet:IGNORED:                             Begin exception details.
Mar 21 20:06:05 localhost org.fedoraproject.Anaconda.Modules.Storage[2271]: DEBUG:blivet:IGNORED:                                 Traceback (most recent call last):
Mar 21 20:06:05 localhost org.fedoraproject.Anaconda.Modules.Storage[2271]: DEBUG:blivet:IGNORED:                                   File "/usr/lib/python3.7/site-packages/blivet/safe_dbus.py", line 147, in call_sync
Mar 21 20:06:05 localhost org.fedoraproject.Anaconda.Modules.Storage[2271]: DEBUG:blivet:IGNORED:                                     DEFAULT_DBUS_TIMEOUT, None)
Mar 21 20:06:05 localhost org.fedoraproject.Anaconda.Modules.Storage[2271]: DEBUG:blivet:IGNORED:                                 gi.repository.GLib.GError: g-io-error-quark: GDBus.Error:org.freedesktop.UDisks2.Error.ISCSI.NoFirmware: No firmware found (36)
Mar 21 20:06:05 localhost org.fedoraproject.Anaconda.Modules.Storage[2271]: DEBUG:blivet:IGNORED:                                 During handling of the above exception, another exception occurred:
Mar 21 20:06:05 localhost org.fedoraproject.Anaconda.Modules.Storage[2271]: DEBUG:blivet:IGNORED:                                 Traceback (most recent call last):
Mar 21 20:06:05 localhost org.fedoraproject.Anaconda.Modules.Storage[2271]: DEBUG:blivet:IGNORED:                                   File "/usr/lib/python3.7/site-packages/blivet/iscsi.py", line 160, in __init__
Mar 21 20:06:05 localhost org.fedoraproject.Anaconda.Modules.Storage[2271]: DEBUG:blivet:IGNORED:                                     initiatorname = self._call_initiator_method("GetFirmwareInitiatorName")[0]
Mar 21 20:06:05 localhost org.fedoraproject.Anaconda.Modules.Storage[2271]: DEBUG:blivet:IGNORED:                                   File "/usr/lib/python3.7/site-packages/blivet/util.py", line 1095, in decorated
Mar 21 20:06:05 localhost org.fedoraproject.Anaconda.Modules.Storage[2271]: DEBUG:blivet:IGNORED:                                     return fn(*args, **kwargs)
Mar 21 20:06:05 localhost org.fedoraproject.Anaconda.Modules.Storage[2271]: DEBUG:blivet:IGNORED:                                   File "/usr/lib/python3.7/site-packages/blivet/iscsi.py", line 196, in _call_initiator_method
Mar 21 20:06:05 localhost org.fedoraproject.Anaconda.Modules.Storage[2271]: DEBUG:blivet:IGNORED:                                     connection=self._connection)
Mar 21 20:06:05 localhost org.fedoraproject.Anaconda.Modules.Storage[2271]: DEBUG:blivet:IGNORED:                                   File "/usr/lib/python3.7/site-packages/blivet/safe_dbus.py", line 151, in call_sync
Mar 21 20:06:05 localhost org.fedoraproject.Anaconda.Modules.Storage[2271]: DEBUG:blivet:IGNORED:                                     raise DBusCallError(msg)
Mar 21 20:06:05 localhost org.fedoraproject.Anaconda.Modules.Storage[2271]: DEBUG:blivet:IGNORED:                                 blivet.safe_dbus.DBusCallError: Failed to call GetFirmwareInitiatorName method on /org/freedesktop/UDisks2/Manager with None arguments: GDBus.Error:org.freedesktop.UDisks2.Error.ISCSI.NoFirmware: No firmware found
Mar 21 20:06:05 localhost org.fedoraproject.Anaconda.Modules.Storage[2271]: DEBUG:blivet:IGNORED:                             End exception details.
===
Comment 5 Adam Williamson 2019-03-22 15:13:17 UTC 
This is probably the same as https://bugzilla.redhat.com/show_bug.cgi?id=1689037 , I think. If you get a better backtrace we could confirm it.
Comment 6 Michel Normand 2019-03-29 10:28:34 UTC 

*** This bug has been marked as a duplicate of bug 1689037 ***