Bug 1008018 - mod_wsgi crash during startup
Summary: mod_wsgi crash during startup
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: mod_wsgi
Version: 6.4
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Jan Kaluža
QA Contact: Jan Kepler
URL:
Whiteboard:
: 1101652 1126987 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-09-13 19:39 UTC by Scott Poore
Modified: 2014-10-14 08:30 UTC (History)
8 users (show)

Fixed In Version: mod_wsgi-3.2-7.el6
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-10-14 08:30:12 UTC
Target Upstream Version:


Attachments (Terms of Use)
abrt crash output (55.54 KB, text/plain)
2013-09-13 19:40 UTC, Scott Poore
no flags Details
abrt crash dump dir from httpd crash (4.24 MB, application/x-tar)
2013-09-13 19:41 UTC, Scott Poore
no flags Details
proposed patch (1.25 KB, patch)
2013-09-17 06:48 UTC, Jan Kaluža
no flags Details | Diff
abrt backtrace from rhel6.6 crash (151.21 KB, text/plain)
2014-08-08 15:56 UTC, Scott Poore
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2014:1612 normal SHIPPED_LIVE mod_wsgi bug fix update 2014-10-14 01:39:23 UTC

Description Scott Poore 2013-09-13 19:39:53 UTC
Description of problem:

abrt is picking up an httpd crash during ipa-replica-install.  

Version-Release number of selected component (if applicable):
httpd-2.2.15-26.el6
ipa-server-3.0.0-25.el6.x86_64

How reproducible:
unknown. but, seems to be occurring very frequently on automated IPA tests.

Steps to Reproduce:

On MASTER:

1.  Install IPA Master

ipa-server-install --setup-dns --forwarder=$DNSFORWARD --hostname=$hostname_s.$DOMAIN -r $RELM -n $DOMAIN -p $ADMINPW -P $ADMINPW -a $ADMINPW -U

2.  Prepare to install IPA Replica

ipa-replica-prepare -p $ADMINPW --ip-address=$ipaddr $hostname_s.$DOMAIN

On REPLICA:

3.  Copy replica info file to replica

sftp -o StrictHostKeyChecking=no root@$MYMASTER:/var/lib/ipa/replica-info-$hostname_s.$DOMAIN.gpg /dev/shm/

4.  Install Replica

ipa-replica-install -U --setup-ca --setup-dns --forwarder=$DNSFORWARD -w $ADMINPW -p $ADMINPW /dev/shm/replica-info-$hostname_s.$DOMAIN.gpg


Actual results:
httpd is crashing and throwing errors but, is running after ipa-replica-install

Expected results:
no crash reported by abrt

Additional info:

From /var/log/messages, it appears to be happening shortly after ipa-replica-install sets some related httpd selinux booleans:

Sep 13 14:40:50 ipaqavmg setsebool: The httpd_can_network_connect policy boolean was changed to on by root
Sep 13 14:40:50 ipaqavmg setsebool: The httpd_manage_ipa policy boolean was changed to on by root
Sep 13 14:40:53 ipaqavmg kernel: httpd[17189]: segfault at 30 ip 00007f92e7ec27c7 sp 00007fffdb81a550 e
rror 4 in libapr-1.so.0.3.9[7f92e7eae000+2b000]
Sep 13 14:40:53 ipaqavmg kernel: httpd[17190]: segfault at 30 ip 00007f92e7ec27c7 sp 00007fffdb819550 error 4 in libapr-1.so.0.3.9[7f92e7eae000+2b000]
Sep 13 14:40:53 ipaqavmg abrt[17222]: Not saving repeating crash in '/usr/sbin/httpd'
Sep 13 14:40:54 ipaqavmg abrt[17220]: Saved core dump of pid 17189 (/usr/sbin/httpd) to /var/spool/abrt/ccpp-2013-09-13-14:40:53-17189 (76107776 bytes)

From /var/log/ipareplica-install.log it looks like it's dying/crashing after the restart?

2013-09-13T18:40:12Z DEBUG   [10/12]: configuring SELinux for httpd
2013-09-13T18:40:12Z DEBUG args=/usr/sbin/selinuxenabled
2013-09-13T18:40:12Z DEBUG stdout=
2013-09-13T18:40:12Z DEBUG stderr=
2013-09-13T18:40:12Z DEBUG args=/usr/sbin/getsebool httpd_can_network_connect
2013-09-13T18:40:12Z DEBUG stdout=httpd_can_network_connect --> off

2013-09-13T18:40:12Z DEBUG stderr=
2013-09-13T18:40:12Z DEBUG Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state'
2013-09-13T18:40:12Z DEBUG args=/usr/sbin/getsebool httpd_manage_ipa
2013-09-13T18:40:12Z DEBUG stdout=httpd_manage_ipa --> off

2013-09-13T18:40:12Z DEBUG stderr=
2013-09-13T18:40:12Z DEBUG Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state'
2013-09-13T18:40:50Z DEBUG args=/usr/sbin/setsebool -P httpd_can_network_connect=on httpd_manage_ipa=on
2013-09-13T18:40:50Z DEBUG stdout=
2013-09-13T18:40:50Z DEBUG stderr=
2013-09-13T18:40:50Z DEBUG   duration: 38 seconds
2013-09-13T18:40:50Z DEBUG   [11/12]: restarting httpd
2013-09-13T18:40:51Z DEBUG args=/sbin/service httpd status 
2013-09-13T18:40:51Z DEBUG stdout=httpd is stopped

2013-09-13T18:40:51Z DEBUG stderr=
2013-09-13T18:40:51Z DEBUG Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state'
2013-09-13T18:40:51Z DEBUG args=/sbin/service httpd restart 
2013-09-13T18:40:51Z DEBUG stdout=Stopping httpd: [FAILED]
Starting httpd: [  OK  ]

2013-09-13T18:40:51Z DEBUG stderr=
2013-09-13T18:40:51Z DEBUG args=/sbin/service httpd status 
2013-09-13T18:40:51Z DEBUG stdout=httpd dead but subsys locked

2013-09-13T18:40:51Z DEBUG stderr=
2013-09-13T18:40:51Z DEBUG   duration: 0 seconds
2013-09-13T18:40:51Z DEBUG   [12/12]: configuring httpd to start on boot
2013-09-13T18:40:51Z DEBUG args=/sbin/service httpd status 
2013-09-13T18:40:51Z DEBUG stdout=httpd dead but subsys locked

2013-09-13T18:40:51Z DEBUG stderr=
2013-09-13T18:40:51Z DEBUG Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state'
2013-09-13T18:40:52Z DEBUG args=/sbin/chkconfig httpd off
2013-09-13T18:40:52Z DEBUG stdout=
2013-09-13T18:40:52Z DEBUG stderr=
2013-09-13T18:40:52Z DEBUG   duration: 0 seconds
2013-09-13T18:40:52Z DEBUG Done configuring the web interface (httpd).

Comment 1 Scott Poore 2013-09-13 19:40:36 UTC
Created attachment 797515 [details]
abrt crash output

Comment 2 Scott Poore 2013-09-13 19:41:08 UTC
Created attachment 797518 [details]
abrt crash dump dir from httpd crash

Comment 3 Scott Poore 2013-09-13 20:50:32 UTC
FYI, I was able to reproduce this on local VMs with the following:

# ON MASTER:

service iptables stop
service ip6tables stop
yum -y install ipa-server bind-dyndb-ldap
echo "$(hostname -i) $(hostname -s).testrelm.com" >> /etc/hosts
sed -i "s/^\(HOSTNAME=\).*$/\1$(hostname -s).testrelm.com/" \
    /etc/sysconfig/network
hostname $(hostname -s).testrelm.com
ipa-server-install --setup-dns --forwarder=192.168.122.1 \
    --hostname=$(hostname -s).testrelm.com -r TESTRELM.COM \
    -n testrelm.com -p Secret123 -P Secret123 -a Secret123 -U
ipa-replica-prepare -p Secret123 --ip-address=192.168.122.62 rhel6-2.testrelm.com

# ON REPLICA:

export MYMASTER=rhel6-1.testrelm.com
export MYMASTER_IP=192.168.122.61
service iptables stop
service ip6tables stop
yum -y install ipa-server bind-dyndb-ldap
echo "$(hostname -i) $(hostname -s).testrelm.com" >> /etc/hosts
sed -i "s/^\(HOSTNAME=\).*$/\1$(hostname -s).testrelm.com/" \
    /etc/sysconfig/network
hostname $(hostname -s).testrelm.com
cp /etc/resolv.conf /etc/resolv.conf.backup1
sed -i '/nameserver/d' /etc/resolv.conf
echo "nameserver $MYMASTER_IP" >> /etc/resolv.conf
sftp root@$MYMASTER:/var/lib/ipa/replica-info-$(hostname -s).testrelm.com.gpg \
    /dev/shm
ipa-replica-install -U --setup-ca --setup-dns \
    --forwarder=192.168.122.1 -w Secret123 -p Secret123 \
    /dev/shm/replica-info-$(hostname -s).testrelm.com.gpg

Comment 5 Jan Kaluža 2013-09-17 06:48:35 UTC
Created attachment 798635 [details]
proposed patch

Moves apr_signal after pipe creation, so when signal is handled, it has all the variables initialized properly in handler.

Comment 7 Joe Orton 2013-09-17 20:30:45 UTC
Scott can you retest with this scratch build of mod_wsgi?

https://brewweb.devel.redhat.com/taskinfo?taskID=6296160

Comment 8 Scott Poore 2013-09-18 02:59:41 UTC
That does seemed to have fixed the issue from the manual and automated tests I ran today.   I didn't see a crash from either.

Comment 15 Rob Crittenden 2014-05-28 12:41:43 UTC
*** Bug 1101652 has been marked as a duplicate of this bug. ***

Comment 16 Scott Poore 2014-08-08 15:55:18 UTC
I believe I am seeing this crash again with this version of httpd.  Unfortunately, the automated test where I saw it no longer has the servers available to log in and grab more log files.  

httpd-2.2.15-38.el6.x86_64

Is this scheduled to be fixed in RHEL6.6?

I'll also attach the latest abrt report we got on this one to confirm/deny that what I'm seeing is indeed the same.

Thanks,
Scott

Comment 17 Scott Poore 2014-08-08 15:56:47 UTC
Created attachment 925214 [details]
abrt backtrace from rhel6.6 crash

Comment 21 Jan Kaluža 2014-08-11 11:11:09 UTC
*** Bug 1126987 has been marked as a duplicate of this bug. ***

Comment 29 errata-xmlrpc 2014-10-14 08:30:12 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1612.html


Note You need to log in before you can comment on or make changes to this bug.