Bug 1008018

Summary: mod_wsgi crash during startup
Product: Red Hat Enterprise Linux 6 Reporter: Scott Poore <spoore>
Component: mod_wsgiAssignee: Jan Kaluža <jkaluza>
Status: CLOSED ERRATA QA Contact: Jan Kepler <jkejda>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 6.4CC: jgalipea, jkaluza, jkejda, jorton, mkosek, rcritten, spoore, tlavigne
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: mod_wsgi-3.2-7.el6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-10-14 08:30:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
abrt crash output
none
abrt crash dump dir from httpd crash
none
proposed patch
none
abrt backtrace from rhel6.6 crash none

Description Scott Poore 2013-09-13 19:39:53 UTC 
Description of problem:

abrt is picking up an httpd crash during ipa-replica-install.  

Version-Release number of selected component (if applicable):
httpd-2.2.15-26.el6
ipa-server-3.0.0-25.el6.x86_64

How reproducible:
unknown. but, seems to be occurring very frequently on automated IPA tests.

Steps to Reproduce:

On MASTER:

1.  Install IPA Master

ipa-server-install --setup-dns --forwarder=$DNSFORWARD --hostname=$hostname_s.$DOMAIN -r $RELM -n $DOMAIN -p $ADMINPW -P $ADMINPW -a $ADMINPW -U

2.  Prepare to install IPA Replica

ipa-replica-prepare -p $ADMINPW --ip-address=$ipaddr $hostname_s.$DOMAIN

On REPLICA:

3.  Copy replica info file to replica

sftp -o StrictHostKeyChecking=no root@$MYMASTER:/var/lib/ipa/replica-info-$hostname_s.$DOMAIN.gpg /dev/shm/

4.  Install Replica

ipa-replica-install -U --setup-ca --setup-dns --forwarder=$DNSFORWARD -w $ADMINPW -p $ADMINPW /dev/shm/replica-info-$hostname_s.$DOMAIN.gpg


Actual results:
httpd is crashing and throwing errors but, is running after ipa-replica-install

Expected results:
no crash reported by abrt

Additional info:

From /var/log/messages, it appears to be happening shortly after ipa-replica-install sets some related httpd selinux booleans:

Sep 13 14:40:50 ipaqavmg setsebool: The httpd_can_network_connect policy boolean was changed to on by root
Sep 13 14:40:50 ipaqavmg setsebool: The httpd_manage_ipa policy boolean was changed to on by root
Sep 13 14:40:53 ipaqavmg kernel: httpd[17189]: segfault at 30 ip 00007f92e7ec27c7 sp 00007fffdb81a550 e
rror 4 in libapr-1.so.0.3.9[7f92e7eae000+2b000]
Sep 13 14:40:53 ipaqavmg kernel: httpd[17190]: segfault at 30 ip 00007f92e7ec27c7 sp 00007fffdb819550 error 4 in libapr-1.so.0.3.9[7f92e7eae000+2b000]
Sep 13 14:40:53 ipaqavmg abrt[17222]: Not saving repeating crash in '/usr/sbin/httpd'
Sep 13 14:40:54 ipaqavmg abrt[17220]: Saved core dump of pid 17189 (/usr/sbin/httpd) to /var/spool/abrt/ccpp-2013-09-13-14:40:53-17189 (76107776 bytes)

From /var/log/ipareplica-install.log it looks like it's dying/crashing after the restart?

2013-09-13T18:40:12Z DEBUG   [10/12]: configuring SELinux for httpd
2013-09-13T18:40:12Z DEBUG args=/usr/sbin/selinuxenabled
2013-09-13T18:40:12Z DEBUG stdout=
2013-09-13T18:40:12Z DEBUG stderr=
2013-09-13T18:40:12Z DEBUG args=/usr/sbin/getsebool httpd_can_network_connect
2013-09-13T18:40:12Z DEBUG stdout=httpd_can_network_connect --> off

2013-09-13T18:40:12Z DEBUG stderr=
2013-09-13T18:40:12Z DEBUG Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state'
2013-09-13T18:40:12Z DEBUG args=/usr/sbin/getsebool httpd_manage_ipa
2013-09-13T18:40:12Z DEBUG stdout=httpd_manage_ipa --> off

2013-09-13T18:40:12Z DEBUG stderr=
2013-09-13T18:40:12Z DEBUG Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state'
2013-09-13T18:40:50Z DEBUG args=/usr/sbin/setsebool -P httpd_can_network_connect=on httpd_manage_ipa=on
2013-09-13T18:40:50Z DEBUG stdout=
2013-09-13T18:40:50Z DEBUG stderr=
2013-09-13T18:40:50Z DEBUG   duration: 38 seconds
2013-09-13T18:40:50Z DEBUG   [11/12]: restarting httpd
2013-09-13T18:40:51Z DEBUG args=/sbin/service httpd status 
2013-09-13T18:40:51Z DEBUG stdout=httpd is stopped

2013-09-13T18:40:51Z DEBUG stderr=
2013-09-13T18:40:51Z DEBUG Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state'
2013-09-13T18:40:51Z DEBUG args=/sbin/service httpd restart 
2013-09-13T18:40:51Z DEBUG stdout=Stopping httpd: [FAILED]
Starting httpd: [  OK  ]

2013-09-13T18:40:51Z DEBUG stderr=
2013-09-13T18:40:51Z DEBUG args=/sbin/service httpd status 
2013-09-13T18:40:51Z DEBUG stdout=httpd dead but subsys locked

2013-09-13T18:40:51Z DEBUG stderr=
2013-09-13T18:40:51Z DEBUG   duration: 0 seconds
2013-09-13T18:40:51Z DEBUG   [12/12]: configuring httpd to start on boot
2013-09-13T18:40:51Z DEBUG args=/sbin/service httpd status 
2013-09-13T18:40:51Z DEBUG stdout=httpd dead but subsys locked

2013-09-13T18:40:51Z DEBUG stderr=
2013-09-13T18:40:51Z DEBUG Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state'
2013-09-13T18:40:52Z DEBUG args=/sbin/chkconfig httpd off
2013-09-13T18:40:52Z DEBUG stdout=
2013-09-13T18:40:52Z DEBUG stderr=
2013-09-13T18:40:52Z DEBUG   duration: 0 seconds
2013-09-13T18:40:52Z DEBUG Done configuring the web interface (httpd).
Comment 1 Scott Poore 2013-09-13 19:40:36 UTC 
Created attachment 797515 [details]
abrt crash output
Comment 2 Scott Poore 2013-09-13 19:41:08 UTC 
Created attachment 797518 [details]
abrt crash dump dir from httpd crash
Comment 3 Scott Poore 2013-09-13 20:50:32 UTC 
FYI, I was able to reproduce this on local VMs with the following:

# ON MASTER:

service iptables stop
service ip6tables stop
yum -y install ipa-server bind-dyndb-ldap
echo "$(hostname -i) $(hostname -s).testrelm.com" >> /etc/hosts
sed -i "s/^\(HOSTNAME=\).*$/\1$(hostname -s).testrelm.com/" \
    /etc/sysconfig/network
hostname $(hostname -s).testrelm.com
ipa-server-install --setup-dns --forwarder=192.168.122.1 \
    --hostname=$(hostname -s).testrelm.com -r TESTRELM.COM \
    -n testrelm.com -p Secret123 -P Secret123 -a Secret123 -U
ipa-replica-prepare -p Secret123 --ip-address=192.168.122.62 rhel6-2.testrelm.com

# ON REPLICA:

export MYMASTER=rhel6-1.testrelm.com
export MYMASTER_IP=192.168.122.61
service iptables stop
service ip6tables stop
yum -y install ipa-server bind-dyndb-ldap
echo "$(hostname -i) $(hostname -s).testrelm.com" >> /etc/hosts
sed -i "s/^\(HOSTNAME=\).*$/\1$(hostname -s).testrelm.com/" \
    /etc/sysconfig/network
hostname $(hostname -s).testrelm.com
cp /etc/resolv.conf /etc/resolv.conf.backup1
sed -i '/nameserver/d' /etc/resolv.conf
echo "nameserver $MYMASTER_IP" >> /etc/resolv.conf
sftp root@$MYMASTER:/var/lib/ipa/replica-info-$(hostname -s).testrelm.com.gpg \
    /dev/shm
ipa-replica-install -U --setup-ca --setup-dns \
    --forwarder=192.168.122.1 -w Secret123 -p Secret123 \
    /dev/shm/replica-info-$(hostname -s).testrelm.com.gpg
Comment 5 Jan Kaluža 2013-09-17 06:48:35 UTC 
Created attachment 798635 [details]
proposed patch

Moves apr_signal after pipe creation, so when signal is handled, it has all the variables initialized properly in handler.
Comment 7 Joe Orton 2013-09-17 20:30:45 UTC 
Scott can you retest with this scratch build of mod_wsgi?

https://brewweb.devel.redhat.com/taskinfo?taskID=6296160
Comment 8 Scott Poore 2013-09-18 02:59:41 UTC 
That does seemed to have fixed the issue from the manual and automated tests I ran today.   I didn't see a crash from either.
Comment 15 Rob Crittenden 2014-05-28 12:41:43 UTC 
*** Bug 1101652 has been marked as a duplicate of this bug. ***
Comment 16 Scott Poore 2014-08-08 15:55:18 UTC 
I believe I am seeing this crash again with this version of httpd.  Unfortunately, the automated test where I saw it no longer has the servers available to log in and grab more log files.  

httpd-2.2.15-38.el6.x86_64

Is this scheduled to be fixed in RHEL6.6?

I'll also attach the latest abrt report we got on this one to confirm/deny that what I'm seeing is indeed the same.

Thanks,
Scott
Comment 17 Scott Poore 2014-08-08 15:56:47 UTC 
Created attachment 925214 [details]
abrt backtrace from rhel6.6 crash
Comment 21 Jan Kaluža 2014-08-11 11:11:09 UTC 
*** Bug 1126987 has been marked as a duplicate of this bug. ***
Comment 29 errata-xmlrpc 2014-10-14 08:30:12 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1612.html