Bug 1008018 - mod_wsgi crash during startup
mod_wsgi crash during startup
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: mod_wsgi (Show other bugs)
6.4
Unspecified Unspecified
unspecified Severity unspecified
: rc
: ---
Assigned To: Jan Kaluža
Jan Kepler
:
: 1101652 1126987 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-09-13 15:39 EDT by Scott Poore
Modified: 2014-10-14 04:30 EDT (History)
8 users (show)

See Also:
Fixed In Version: mod_wsgi-3.2-7.el6
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-10-14 04:30:12 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
abrt crash output (55.54 KB, text/plain)
2013-09-13 15:40 EDT, Scott Poore
no flags Details
abrt crash dump dir from httpd crash (4.24 MB, application/x-tar)
2013-09-13 15:41 EDT, Scott Poore
no flags Details
proposed patch (1.25 KB, patch)
2013-09-17 02:48 EDT, Jan Kaluža
no flags Details | Diff
abrt backtrace from rhel6.6 crash (151.21 KB, text/plain)
2014-08-08 11:56 EDT, Scott Poore
no flags Details

  None (edit)
Description Scott Poore 2013-09-13 15:39:53 EDT
Description of problem:

abrt is picking up an httpd crash during ipa-replica-install.  

Version-Release number of selected component (if applicable):
httpd-2.2.15-26.el6
ipa-server-3.0.0-25.el6.x86_64

How reproducible:
unknown. but, seems to be occurring very frequently on automated IPA tests.

Steps to Reproduce:

On MASTER:

1.  Install IPA Master

ipa-server-install --setup-dns --forwarder=$DNSFORWARD --hostname=$hostname_s.$DOMAIN -r $RELM -n $DOMAIN -p $ADMINPW -P $ADMINPW -a $ADMINPW -U

2.  Prepare to install IPA Replica

ipa-replica-prepare -p $ADMINPW --ip-address=$ipaddr $hostname_s.$DOMAIN

On REPLICA:

3.  Copy replica info file to replica

sftp -o StrictHostKeyChecking=no root@$MYMASTER:/var/lib/ipa/replica-info-$hostname_s.$DOMAIN.gpg /dev/shm/

4.  Install Replica

ipa-replica-install -U --setup-ca --setup-dns --forwarder=$DNSFORWARD -w $ADMINPW -p $ADMINPW /dev/shm/replica-info-$hostname_s.$DOMAIN.gpg


Actual results:
httpd is crashing and throwing errors but, is running after ipa-replica-install

Expected results:
no crash reported by abrt

Additional info:

From /var/log/messages, it appears to be happening shortly after ipa-replica-install sets some related httpd selinux booleans:

Sep 13 14:40:50 ipaqavmg setsebool: The httpd_can_network_connect policy boolean was changed to on by root
Sep 13 14:40:50 ipaqavmg setsebool: The httpd_manage_ipa policy boolean was changed to on by root
Sep 13 14:40:53 ipaqavmg kernel: httpd[17189]: segfault at 30 ip 00007f92e7ec27c7 sp 00007fffdb81a550 e
rror 4 in libapr-1.so.0.3.9[7f92e7eae000+2b000]
Sep 13 14:40:53 ipaqavmg kernel: httpd[17190]: segfault at 30 ip 00007f92e7ec27c7 sp 00007fffdb819550 error 4 in libapr-1.so.0.3.9[7f92e7eae000+2b000]
Sep 13 14:40:53 ipaqavmg abrt[17222]: Not saving repeating crash in '/usr/sbin/httpd'
Sep 13 14:40:54 ipaqavmg abrt[17220]: Saved core dump of pid 17189 (/usr/sbin/httpd) to /var/spool/abrt/ccpp-2013-09-13-14:40:53-17189 (76107776 bytes)

From /var/log/ipareplica-install.log it looks like it's dying/crashing after the restart?

2013-09-13T18:40:12Z DEBUG   [10/12]: configuring SELinux for httpd
2013-09-13T18:40:12Z DEBUG args=/usr/sbin/selinuxenabled
2013-09-13T18:40:12Z DEBUG stdout=
2013-09-13T18:40:12Z DEBUG stderr=
2013-09-13T18:40:12Z DEBUG args=/usr/sbin/getsebool httpd_can_network_connect
2013-09-13T18:40:12Z DEBUG stdout=httpd_can_network_connect --> off

2013-09-13T18:40:12Z DEBUG stderr=
2013-09-13T18:40:12Z DEBUG Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state'
2013-09-13T18:40:12Z DEBUG args=/usr/sbin/getsebool httpd_manage_ipa
2013-09-13T18:40:12Z DEBUG stdout=httpd_manage_ipa --> off

2013-09-13T18:40:12Z DEBUG stderr=
2013-09-13T18:40:12Z DEBUG Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state'
2013-09-13T18:40:50Z DEBUG args=/usr/sbin/setsebool -P httpd_can_network_connect=on httpd_manage_ipa=on
2013-09-13T18:40:50Z DEBUG stdout=
2013-09-13T18:40:50Z DEBUG stderr=
2013-09-13T18:40:50Z DEBUG   duration: 38 seconds
2013-09-13T18:40:50Z DEBUG   [11/12]: restarting httpd
2013-09-13T18:40:51Z DEBUG args=/sbin/service httpd status 
2013-09-13T18:40:51Z DEBUG stdout=httpd is stopped

2013-09-13T18:40:51Z DEBUG stderr=
2013-09-13T18:40:51Z DEBUG Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state'
2013-09-13T18:40:51Z DEBUG args=/sbin/service httpd restart 
2013-09-13T18:40:51Z DEBUG stdout=Stopping httpd: [FAILED]
Starting httpd: [  OK  ]

2013-09-13T18:40:51Z DEBUG stderr=
2013-09-13T18:40:51Z DEBUG args=/sbin/service httpd status 
2013-09-13T18:40:51Z DEBUG stdout=httpd dead but subsys locked

2013-09-13T18:40:51Z DEBUG stderr=
2013-09-13T18:40:51Z DEBUG   duration: 0 seconds
2013-09-13T18:40:51Z DEBUG   [12/12]: configuring httpd to start on boot
2013-09-13T18:40:51Z DEBUG args=/sbin/service httpd status 
2013-09-13T18:40:51Z DEBUG stdout=httpd dead but subsys locked

2013-09-13T18:40:51Z DEBUG stderr=
2013-09-13T18:40:51Z DEBUG Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state'
2013-09-13T18:40:52Z DEBUG args=/sbin/chkconfig httpd off
2013-09-13T18:40:52Z DEBUG stdout=
2013-09-13T18:40:52Z DEBUG stderr=
2013-09-13T18:40:52Z DEBUG   duration: 0 seconds
2013-09-13T18:40:52Z DEBUG Done configuring the web interface (httpd).
Comment 1 Scott Poore 2013-09-13 15:40:36 EDT
Created attachment 797515 [details]
abrt crash output
Comment 2 Scott Poore 2013-09-13 15:41:08 EDT
Created attachment 797518 [details]
abrt crash dump dir from httpd crash
Comment 3 Scott Poore 2013-09-13 16:50:32 EDT
FYI, I was able to reproduce this on local VMs with the following:

# ON MASTER:

service iptables stop
service ip6tables stop
yum -y install ipa-server bind-dyndb-ldap
echo "$(hostname -i) $(hostname -s).testrelm.com" >> /etc/hosts
sed -i "s/^\(HOSTNAME=\).*$/\1$(hostname -s).testrelm.com/" \
    /etc/sysconfig/network
hostname $(hostname -s).testrelm.com
ipa-server-install --setup-dns --forwarder=192.168.122.1 \
    --hostname=$(hostname -s).testrelm.com -r TESTRELM.COM \
    -n testrelm.com -p Secret123 -P Secret123 -a Secret123 -U
ipa-replica-prepare -p Secret123 --ip-address=192.168.122.62 rhel6-2.testrelm.com

# ON REPLICA:

export MYMASTER=rhel6-1.testrelm.com
export MYMASTER_IP=192.168.122.61
service iptables stop
service ip6tables stop
yum -y install ipa-server bind-dyndb-ldap
echo "$(hostname -i) $(hostname -s).testrelm.com" >> /etc/hosts
sed -i "s/^\(HOSTNAME=\).*$/\1$(hostname -s).testrelm.com/" \
    /etc/sysconfig/network
hostname $(hostname -s).testrelm.com
cp /etc/resolv.conf /etc/resolv.conf.backup1
sed -i '/nameserver/d' /etc/resolv.conf
echo "nameserver $MYMASTER_IP" >> /etc/resolv.conf
sftp root@$MYMASTER:/var/lib/ipa/replica-info-$(hostname -s).testrelm.com.gpg \
    /dev/shm
ipa-replica-install -U --setup-ca --setup-dns \
    --forwarder=192.168.122.1 -w Secret123 -p Secret123 \
    /dev/shm/replica-info-$(hostname -s).testrelm.com.gpg
Comment 5 Jan Kaluža 2013-09-17 02:48:35 EDT
Created attachment 798635 [details]
proposed patch

Moves apr_signal after pipe creation, so when signal is handled, it has all the variables initialized properly in handler.
Comment 7 Joe Orton 2013-09-17 16:30:45 EDT
Scott can you retest with this scratch build of mod_wsgi?

https://brewweb.devel.redhat.com/taskinfo?taskID=6296160
Comment 8 Scott Poore 2013-09-17 22:59:41 EDT
That does seemed to have fixed the issue from the manual and automated tests I ran today.   I didn't see a crash from either.
Comment 15 Rob Crittenden 2014-05-28 08:41:43 EDT
*** Bug 1101652 has been marked as a duplicate of this bug. ***
Comment 16 Scott Poore 2014-08-08 11:55:18 EDT
I believe I am seeing this crash again with this version of httpd.  Unfortunately, the automated test where I saw it no longer has the servers available to log in and grab more log files.  

httpd-2.2.15-38.el6.x86_64

Is this scheduled to be fixed in RHEL6.6?

I'll also attach the latest abrt report we got on this one to confirm/deny that what I'm seeing is indeed the same.

Thanks,
Scott
Comment 17 Scott Poore 2014-08-08 11:56:47 EDT
Created attachment 925214 [details]
abrt backtrace from rhel6.6 crash
Comment 21 Jan Kaluža 2014-08-11 07:11:09 EDT
*** Bug 1126987 has been marked as a duplicate of this bug. ***
Comment 29 errata-xmlrpc 2014-10-14 04:30:12 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1612.html

Note You need to log in before you can comment on or make changes to this bug.