Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1349308

Summary: RHEL-7.3-20160621.n.0 installation sporadically stalling
Product: Red Hat Enterprise Linux 7 Reporter: Jan Stancek <jstancek>
Component: kexec-toolsAssignee: Dave Young <ruyang>
Status: CLOSED ERRATA QA Contact: Qiao Zhao <qzhao>
Severity: high Docs Contact:
Priority: unspecified    
Version: 7.3CC: bhe, bpeck, cye, jburke, jstancek, kdump-team-bugs, mbanas, pbunyan, pholica, qzhao, ruyang
Target Milestone: alphaKeywords: Regression
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: kexec-tools-2.0.7-44.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-11-04 06:56:23 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
anaconda.log
none
console.log
none
ifcfg.log
none
ks.cfg
none
program.log
none
storage.log
none
sys.log
none
do not use total_mem function from ananconda lib
none
do not use total_mem function from ananconda lib none

Description Jan Stancek 2016-06-23 08:11:59 UTC
Description of problem:
RHEL-7.3-20160621.n.0 is sporadically stalling and installation never completes:
-------------------------------------------------------------------------------
01:32:59 Running pre-installation scripts      
  
   
+ command -v curl
+ curl --help
+ grep -q '.*--retry'
+ fetch - http://lab-02.rhts.eng.bos.redhat.com:8000/install_start/2817286
+ curl --retry 20 --remote-time -o - http://lab-02.rhts.eng.bos.redhat.com:8000/install_start/2817286
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0 100     4  100     4    0     0     41      0 --:--:-- --:--:-- --:--:--    40 100     4  100     4    0     0     41      0 --:--:-- --:--:-- --:--:--    40
True+ fetch /tmp/anamon http://lab-02.rhts.eng.bos.redhat.com/beaker/anamon
+ curl --retry 20 --remote-time -o /tmp/anamon http://lab-02.rhts.eng.bos.redhat.com/beaker/anamon
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0 100  8757  100  8757    0     0  1367k      0 --:--:-- --:--:-- --:--:-- 1425k
+ python /tmp/anamon --recipe-id 2817286 --xmlrpc-url http://lab-02.rhts.eng.bos.redhat.com:8000/RPC2
      
  
   
01:33:00 Not asking for VNC because of an automated install      
  
   
      
  
   
01:33:00 Not asking for VNC because text mode was explicitly asked for in kickstart      
  
   
[-- MARK -- Thu Jun 23 01:35:00 2016] 
[-- MARK -- Thu Jun 23 01:40:00 2016] 
[-- MARK -- Thu Jun 23 01:45:00 2016] 
[-- MARK -- Thu Jun 23 01:50:00 2016] 
[-- MARK -- Thu Jun 23 01:55:00 2016] 
[-- MARK -- Thu Jun 23 02:00:00 2016] 
[-- MARK -- Thu Jun 23 02:05:00 2016] 

-------------------------------------------------------------------------------

Version-Release number of selected component (if applicable):
RHEL-7.3-20160621.n.0
anaconda-21.48.22.78-1.el7

How reproducible:
sporadically

Steps to Reproduce:
1. install RHEL-7.3-20160621.n.0

Actual results:
installation never completes

Expected results:
compose can be installed

Additional info:

Comment 1 Jan Stancek 2016-06-23 08:12:31 UTC
Created attachment 1171330 [details]
anaconda.log

Comment 2 Jan Stancek 2016-06-23 08:12:49 UTC
Created attachment 1171331 [details]
console.log

Comment 3 Jan Stancek 2016-06-23 08:13:06 UTC
Created attachment 1171332 [details]
ifcfg.log

Comment 4 Jan Stancek 2016-06-23 08:13:25 UTC
Created attachment 1171333 [details]
ks.cfg

Comment 5 Jan Stancek 2016-06-23 08:13:42 UTC
Created attachment 1171335 [details]
program.log

Comment 6 Jan Stancek 2016-06-23 08:13:59 UTC
Created attachment 1171336 [details]
storage.log

Comment 7 Jan Stancek 2016-06-23 08:14:18 UTC
Created attachment 1171337 [details]
sys.log

Comment 9 Jan Stancek 2016-06-23 10:55:06 UTC
This looks like same deadlock as in Bug 1246048, where import and logging lock get taken in reverse order:

  3    Thread 0x3fff1b23910 (LWP 1700) 0x000003fffcffa52e -> AnaStorageThread
  5    Thread 0x3fffcd8e6f0 (LWP 1623) 0x000003fffcffa52e -> kdump spoke

thread 5
 ...
  import com_redhat_kdump.tui.spokes.%s
   com_redhat_kdump/tui/spokes/kdump.py, line 59, in KdumpSpoke ()
    pyanaconda/isys/__init__.py, line 174, in total_memory
     logging/__init__.py, line 1149, in info 
      threading.py, line 173, in acquire, _RLock__owner=4397806532880 (0x3fff1b23910)

thread 3 (AnaStorageThread)
 ...
  blivet/devices/disk.py, line 560, in __init__ 
   blivet/devices/disk.py, line 89, in __init__ (
    blivet/devices/storage.py, line 142, in __init__
     blivet/devices/storage.py, line 651, in updateSize 
      logging/__init__.py, line 464, in format 
       blivet/size.py, line 292, in humanReadable
        blivet/size.py, line 90, in _xlated_prefixes
         blivet/i18n.py, line 26, in <lambda> (x='mega')
          gettext.py, line 540, in ldgettext
           gettext.py, line 467, in translation
            gettext.py, line 131, in _expand_lang 
             PyImport_ImportModuleLevel (name=0x3fffcd11564 "locale"
             _PyImport_AcquireLock ()

Comment 12 David Shea 2016-06-23 20:31:39 UTC
Thanks for the backtraces, Jan.

kdump-anaconda-addon added a new call to getMemoryBounds in the class init for KdumpSpoke, which, as discussed in bug 1246048, isn't going to work. If the entry needs to have the memory bounds information, could edit_fields be populated in __init__ instead?

Comment 13 Jan Stancek 2016-06-24 08:20:26 UTC
> How reproducible:
> sporadically

My experience so far is 1:20, more so on s390x. Can we expect a fix soon?

Comment 14 Dave Young 2016-06-25 09:52:20 UTC
(In reply to David Shea from comment #12)
> Thanks for the backtraces, Jan.
> 
> kdump-anaconda-addon added a new call to getMemoryBounds in the class init
> for KdumpSpoke, which, as discussed in bug 1246048, isn't going to work. If
> the entry needs to have the memory bounds information, could edit_fields be
> populated in __init__ instead?

I'm not sure how to move edit_fields to __init__, maybe I can switch to parse /proc/meminfo and get MemTotal instead of using the total_memory provided by pyanaconda..

Comment 15 Dave Young 2016-06-25 09:53:50 UTC
(In reply to Jan Stancek from comment #13)
> > How reproducible:
> > sporadically
> 
> My experience so far is 1:20, more so on s390x. Can we expect a fix soon?

Jan, I will work on it this week, but not sure it can be a quick fix because I'm traveling during the week.

Thanks
Dave

Comment 16 Dave Young 2016-06-27 14:47:14 UTC
Created attachment 1172909 [details]
do not use total_mem function from ananconda lib

Comment 17 Dave Young 2016-06-27 14:50:12 UTC
Jan, could you do a test with the fixed addon here, with extra kernel cmdline:
updates=http://people.redhat.com/ruyang/updates.img

Thanks
Dave

Comment 18 Dave Young 2016-06-27 14:57:24 UTC
Created attachment 1172910 [details]
do not use total_mem function from ananconda lib

Comment 19 Jan Stancek 2016-06-27 19:54:10 UTC
(In reply to Dave Young from comment #17)
> Jan, could you do a test with the fixed addon here, with extra kernel
> cmdline:
> updates=http://people.redhat.com/ruyang/updates.img
> 
> Thanks
> Dave

Seems to work OK, no deadlocks after 30 installs.

Comment 20 Dave Young 2016-07-07 07:51:10 UTC
Pushed to internal git:
http://git.engineering.redhat.com/git/users/ruyang/kdump-anaconda-addon/.git/commit/?id=69da0f80a37d50433c7627106137eaa00b49bd94

Bao, can you pull and update the addon tarball in kexec-tools?

Comment 21 Dave Young 2016-07-07 07:51:59 UTC
Qiao, can you provide a qa_ack?

Comment 22 Qiao Zhao 2016-07-08 01:27:47 UTC
(In reply to Dave Young from comment #21)
> Qiao, can you provide a qa_ack?

set qa_ack+.

Comment 26 errata-xmlrpc 2016-11-04 06:56:23 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2466.html