RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1008018 - mod_wsgi crash during startup
Summary: mod_wsgi crash during startup
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: mod_wsgi
Version: 6.4
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Jan Kaluža
QA Contact: Jan Kepler
URL:
Whiteboard:
: 1101652 1126987 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-09-13 19:39 UTC by Scott Poore
Modified: 2014-10-14 08:30 UTC (History)
8 users (show)

Fixed In Version: mod_wsgi-3.2-7.el6
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-10-14 08:30:12 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
abrt crash output (55.54 KB, text/plain)
2013-09-13 19:40 UTC, Scott Poore 		no flags Details
abrt crash dump dir from httpd crash (4.24 MB, application/x-tar)
2013-09-13 19:41 UTC, Scott Poore 		no flags Details
proposed patch (1.25 KB, patch)
2013-09-17 06:48 UTC, Jan Kaluža 		no flags Details | Diff
abrt backtrace from rhel6.6 crash (151.21 KB, text/plain)
2014-08-08 15:56 UTC, Scott Poore 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2014:1612 0 normal SHIPPED_LIVE mod_wsgi bug fix update 2014-10-14 01:39:23 UTC

Description Scott Poore 2013-09-13 19:39:53 UTC 
Description of problem:

abrt is picking up an httpd crash during ipa-replica-install.  

Version-Release number of selected component (if applicable):
httpd-2.2.15-26.el6
ipa-server-3.0.0-25.el6.x86_64

How reproducible:
unknown. but, seems to be occurring very frequently on automated IPA tests.

Steps to Reproduce:

On MASTER:

1.  Install IPA Master

ipa-server-install --setup-dns --forwarder=$DNSFORWARD --hostname=$hostname_s.$DOMAIN -r $RELM -n $DOMAIN -p $ADMINPW -P $ADMINPW -a $ADMINPW -U

2.  Prepare to install IPA Replica

ipa-replica-prepare -p $ADMINPW --ip-address=$ipaddr $hostname_s.$DOMAIN

On REPLICA:

3.  Copy replica info file to replica

sftp -o StrictHostKeyChecking=no root@$MYMASTER:/var/lib/ipa/replica-info-$hostname_s.$DOMAIN.gpg /dev/shm/

4.  Install Replica

ipa-replica-install -U --setup-ca --setup-dns --forwarder=$DNSFORWARD -w $ADMINPW -p $ADMINPW /dev/shm/replica-info-$hostname_s.$DOMAIN.gpg


Actual results:
httpd is crashing and throwing errors but, is running after ipa-replica-install

Expected results:
no crash reported by abrt

Additional info:

From /var/log/messages, it appears to be happening shortly after ipa-replica-install sets some related httpd selinux booleans:

Sep 13 14:40:50 ipaqavmg setsebool: The httpd_can_network_connect policy boolean was changed to on by root
Sep 13 14:40:50 ipaqavmg setsebool: The httpd_manage_ipa policy boolean was changed to on by root
Sep 13 14:40:53 ipaqavmg kernel: httpd[17189]: segfault at 30 ip 00007f92e7ec27c7 sp 00007fffdb81a550 e
rror 4 in libapr-1.so.0.3.9[7f92e7eae000+2b000]
Sep 13 14:40:53 ipaqavmg kernel: httpd[17190]: segfault at 30 ip 00007f92e7ec27c7 sp 00007fffdb819550 error 4 in libapr-1.so.0.3.9[7f92e7eae000+2b000]
Sep 13 14:40:53 ipaqavmg abrt[17222]: Not saving repeating crash in '/usr/sbin/httpd'
Sep 13 14:40:54 ipaqavmg abrt[17220]: Saved core dump of pid 17189 (/usr/sbin/httpd) to /var/spool/abrt/ccpp-2013-09-13-14:40:53-17189 (76107776 bytes)

From /var/log/ipareplica-install.log it looks like it's dying/crashing after the restart?

2013-09-13T18:40:12Z DEBUG   [10/12]: configuring SELinux for httpd
2013-09-13T18:40:12Z DEBUG args=/usr/sbin/selinuxenabled
2013-09-13T18:40:12Z DEBUG stdout=
2013-09-13T18:40:12Z DEBUG stderr=
2013-09-13T18:40:12Z DEBUG args=/usr/sbin/getsebool httpd_can_network_connect
2013-09-13T18:40:12Z DEBUG stdout=httpd_can_network_connect --> off

2013-09-13T18:40:12Z DEBUG stderr=
2013-09-13T18:40:12Z DEBUG Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state'
2013-09-13T18:40:12Z DEBUG args=/usr/sbin/getsebool httpd_manage_ipa
2013-09-13T18:40:12Z DEBUG stdout=httpd_manage_ipa --> off

2013-09-13T18:40:12Z DEBUG stderr=
2013-09-13T18:40:12Z DEBUG Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state'
2013-09-13T18:40:50Z DEBUG args=/usr/sbin/setsebool -P httpd_can_network_connect=on httpd_manage_ipa=on
2013-09-13T18:40:50Z DEBUG stdout=
2013-09-13T18:40:50Z DEBUG stderr=
2013-09-13T18:40:50Z DEBUG   duration: 38 seconds
2013-09-13T18:40:50Z DEBUG   [11/12]: restarting httpd
2013-09-13T18:40:51Z DEBUG args=/sbin/service httpd status 
2013-09-13T18:40:51Z DEBUG stdout=httpd is stopped

2013-09-13T18:40:51Z DEBUG stderr=
2013-09-13T18:40:51Z DEBUG Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state'
2013-09-13T18:40:51Z DEBUG args=/sbin/service httpd restart 
2013-09-13T18:40:51Z DEBUG stdout=Stopping httpd: [FAILED]
Starting httpd: [  OK  ]

2013-09-13T18:40:51Z DEBUG stderr=
2013-09-13T18:40:51Z DEBUG args=/sbin/service httpd status 
2013-09-13T18:40:51Z DEBUG stdout=httpd dead but subsys locked

2013-09-13T18:40:51Z DEBUG stderr=
2013-09-13T18:40:51Z DEBUG   duration: 0 seconds
2013-09-13T18:40:51Z DEBUG   [12/12]: configuring httpd to start on boot
2013-09-13T18:40:51Z DEBUG args=/sbin/service httpd status 
2013-09-13T18:40:51Z DEBUG stdout=httpd dead but subsys locked

2013-09-13T18:40:51Z DEBUG stderr=
2013-09-13T18:40:51Z DEBUG Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state'
2013-09-13T18:40:52Z DEBUG args=/sbin/chkconfig httpd off
2013-09-13T18:40:52Z DEBUG stdout=
2013-09-13T18:40:52Z DEBUG stderr=
2013-09-13T18:40:52Z DEBUG   duration: 0 seconds
2013-09-13T18:40:52Z DEBUG Done configuring the web interface (httpd).
Comment 1 Scott Poore 2013-09-13 19:40:36 UTC 
Created attachment 797515 [details]
abrt crash output
Comment 2 Scott Poore 2013-09-13 19:41:08 UTC 
Created attachment 797518 [details]
abrt crash dump dir from httpd crash
Comment 3 Scott Poore 2013-09-13 20:50:32 UTC 
FYI, I was able to reproduce this on local VMs with the following:

# ON MASTER:

service iptables stop
service ip6tables stop
yum -y install ipa-server bind-dyndb-ldap
echo "$(hostname -i) $(hostname -s).testrelm.com" >> /etc/hosts
sed -i "s/^\(HOSTNAME=\).*$/\1$(hostname -s).testrelm.com/" \
    /etc/sysconfig/network
hostname $(hostname -s).testrelm.com
ipa-server-install --setup-dns --forwarder=192.168.122.1 \
    --hostname=$(hostname -s).testrelm.com -r TESTRELM.COM \
    -n testrelm.com -p Secret123 -P Secret123 -a Secret123 -U
ipa-replica-prepare -p Secret123 --ip-address=192.168.122.62 rhel6-2.testrelm.com

# ON REPLICA:

export MYMASTER=rhel6-1.testrelm.com
export MYMASTER_IP=192.168.122.61
service iptables stop
service ip6tables stop
yum -y install ipa-server bind-dyndb-ldap
echo "$(hostname -i) $(hostname -s).testrelm.com" >> /etc/hosts
sed -i "s/^\(HOSTNAME=\).*$/\1$(hostname -s).testrelm.com/" \
    /etc/sysconfig/network
hostname $(hostname -s).testrelm.com
cp /etc/resolv.conf /etc/resolv.conf.backup1
sed -i '/nameserver/d' /etc/resolv.conf
echo "nameserver $MYMASTER_IP" >> /etc/resolv.conf
sftp root@$MYMASTER:/var/lib/ipa/replica-info-$(hostname -s).testrelm.com.gpg \
    /dev/shm
ipa-replica-install -U --setup-ca --setup-dns \
    --forwarder=192.168.122.1 -w Secret123 -p Secret123 \
    /dev/shm/replica-info-$(hostname -s).testrelm.com.gpg
Comment 5 Jan Kaluža 2013-09-17 06:48:35 UTC 
Created attachment 798635 [details]
proposed patch

Moves apr_signal after pipe creation, so when signal is handled, it has all the variables initialized properly in handler.
Comment 7 Joe Orton 2013-09-17 20:30:45 UTC 
Scott can you retest with this scratch build of mod_wsgi?

https://brewweb.devel.redhat.com/taskinfo?taskID=6296160
Comment 8 Scott Poore 2013-09-18 02:59:41 UTC 
That does seemed to have fixed the issue from the manual and automated tests I ran today.   I didn't see a crash from either.
Comment 15 Rob Crittenden 2014-05-28 12:41:43 UTC 
*** Bug 1101652 has been marked as a duplicate of this bug. ***
Comment 16 Scott Poore 2014-08-08 15:55:18 UTC 
I believe I am seeing this crash again with this version of httpd.  Unfortunately, the automated test where I saw it no longer has the servers available to log in and grab more log files.  

httpd-2.2.15-38.el6.x86_64

Is this scheduled to be fixed in RHEL6.6?

I'll also attach the latest abrt report we got on this one to confirm/deny that what I'm seeing is indeed the same.

Thanks,
Scott
Comment 17 Scott Poore 2014-08-08 15:56:47 UTC 
Created attachment 925214 [details]
abrt backtrace from rhel6.6 crash
Comment 21 Jan Kaluža 2014-08-11 11:11:09 UTC 
*** Bug 1126987 has been marked as a duplicate of this bug. ***
Comment 29 errata-xmlrpc 2014-10-14 08:30:12 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1612.html
Note You need to log in before you can comment on or make changes to this bug.