Hide Forgot
Description of problem: abrt is picking up an httpd crash during ipa-replica-install. Version-Release number of selected component (if applicable): httpd-2.2.15-26.el6 ipa-server-3.0.0-25.el6.x86_64 How reproducible: unknown. but, seems to be occurring very frequently on automated IPA tests. Steps to Reproduce: On MASTER: 1. Install IPA Master ipa-server-install --setup-dns --forwarder=$DNSFORWARD --hostname=$hostname_s.$DOMAIN -r $RELM -n $DOMAIN -p $ADMINPW -P $ADMINPW -a $ADMINPW -U 2. Prepare to install IPA Replica ipa-replica-prepare -p $ADMINPW --ip-address=$ipaddr $hostname_s.$DOMAIN On REPLICA: 3. Copy replica info file to replica sftp -o StrictHostKeyChecking=no root@$MYMASTER:/var/lib/ipa/replica-info-$hostname_s.$DOMAIN.gpg /dev/shm/ 4. Install Replica ipa-replica-install -U --setup-ca --setup-dns --forwarder=$DNSFORWARD -w $ADMINPW -p $ADMINPW /dev/shm/replica-info-$hostname_s.$DOMAIN.gpg Actual results: httpd is crashing and throwing errors but, is running after ipa-replica-install Expected results: no crash reported by abrt Additional info: From /var/log/messages, it appears to be happening shortly after ipa-replica-install sets some related httpd selinux booleans: Sep 13 14:40:50 ipaqavmg setsebool: The httpd_can_network_connect policy boolean was changed to on by root Sep 13 14:40:50 ipaqavmg setsebool: The httpd_manage_ipa policy boolean was changed to on by root Sep 13 14:40:53 ipaqavmg kernel: httpd[17189]: segfault at 30 ip 00007f92e7ec27c7 sp 00007fffdb81a550 e rror 4 in libapr-1.so.0.3.9[7f92e7eae000+2b000] Sep 13 14:40:53 ipaqavmg kernel: httpd[17190]: segfault at 30 ip 00007f92e7ec27c7 sp 00007fffdb819550 error 4 in libapr-1.so.0.3.9[7f92e7eae000+2b000] Sep 13 14:40:53 ipaqavmg abrt[17222]: Not saving repeating crash in '/usr/sbin/httpd' Sep 13 14:40:54 ipaqavmg abrt[17220]: Saved core dump of pid 17189 (/usr/sbin/httpd) to /var/spool/abrt/ccpp-2013-09-13-14:40:53-17189 (76107776 bytes) From /var/log/ipareplica-install.log it looks like it's dying/crashing after the restart? 2013-09-13T18:40:12Z DEBUG [10/12]: configuring SELinux for httpd 2013-09-13T18:40:12Z DEBUG args=/usr/sbin/selinuxenabled 2013-09-13T18:40:12Z DEBUG stdout= 2013-09-13T18:40:12Z DEBUG stderr= 2013-09-13T18:40:12Z DEBUG args=/usr/sbin/getsebool httpd_can_network_connect 2013-09-13T18:40:12Z DEBUG stdout=httpd_can_network_connect --> off 2013-09-13T18:40:12Z DEBUG stderr= 2013-09-13T18:40:12Z DEBUG Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state' 2013-09-13T18:40:12Z DEBUG args=/usr/sbin/getsebool httpd_manage_ipa 2013-09-13T18:40:12Z DEBUG stdout=httpd_manage_ipa --> off 2013-09-13T18:40:12Z DEBUG stderr= 2013-09-13T18:40:12Z DEBUG Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state' 2013-09-13T18:40:50Z DEBUG args=/usr/sbin/setsebool -P httpd_can_network_connect=on httpd_manage_ipa=on 2013-09-13T18:40:50Z DEBUG stdout= 2013-09-13T18:40:50Z DEBUG stderr= 2013-09-13T18:40:50Z DEBUG duration: 38 seconds 2013-09-13T18:40:50Z DEBUG [11/12]: restarting httpd 2013-09-13T18:40:51Z DEBUG args=/sbin/service httpd status 2013-09-13T18:40:51Z DEBUG stdout=httpd is stopped 2013-09-13T18:40:51Z DEBUG stderr= 2013-09-13T18:40:51Z DEBUG Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state' 2013-09-13T18:40:51Z DEBUG args=/sbin/service httpd restart 2013-09-13T18:40:51Z DEBUG stdout=Stopping httpd: [FAILED] Starting httpd: [ OK ] 2013-09-13T18:40:51Z DEBUG stderr= 2013-09-13T18:40:51Z DEBUG args=/sbin/service httpd status 2013-09-13T18:40:51Z DEBUG stdout=httpd dead but subsys locked 2013-09-13T18:40:51Z DEBUG stderr= 2013-09-13T18:40:51Z DEBUG duration: 0 seconds 2013-09-13T18:40:51Z DEBUG [12/12]: configuring httpd to start on boot 2013-09-13T18:40:51Z DEBUG args=/sbin/service httpd status 2013-09-13T18:40:51Z DEBUG stdout=httpd dead but subsys locked 2013-09-13T18:40:51Z DEBUG stderr= 2013-09-13T18:40:51Z DEBUG Saving StateFile to '/var/lib/ipa/sysrestore/sysrestore.state' 2013-09-13T18:40:52Z DEBUG args=/sbin/chkconfig httpd off 2013-09-13T18:40:52Z DEBUG stdout= 2013-09-13T18:40:52Z DEBUG stderr= 2013-09-13T18:40:52Z DEBUG duration: 0 seconds 2013-09-13T18:40:52Z DEBUG Done configuring the web interface (httpd).
Created attachment 797515 [details] abrt crash output
Created attachment 797518 [details] abrt crash dump dir from httpd crash
FYI, I was able to reproduce this on local VMs with the following: # ON MASTER: service iptables stop service ip6tables stop yum -y install ipa-server bind-dyndb-ldap echo "$(hostname -i) $(hostname -s).testrelm.com" >> /etc/hosts sed -i "s/^\(HOSTNAME=\).*$/\1$(hostname -s).testrelm.com/" \ /etc/sysconfig/network hostname $(hostname -s).testrelm.com ipa-server-install --setup-dns --forwarder=192.168.122.1 \ --hostname=$(hostname -s).testrelm.com -r TESTRELM.COM \ -n testrelm.com -p Secret123 -P Secret123 -a Secret123 -U ipa-replica-prepare -p Secret123 --ip-address=192.168.122.62 rhel6-2.testrelm.com # ON REPLICA: export MYMASTER=rhel6-1.testrelm.com export MYMASTER_IP=192.168.122.61 service iptables stop service ip6tables stop yum -y install ipa-server bind-dyndb-ldap echo "$(hostname -i) $(hostname -s).testrelm.com" >> /etc/hosts sed -i "s/^\(HOSTNAME=\).*$/\1$(hostname -s).testrelm.com/" \ /etc/sysconfig/network hostname $(hostname -s).testrelm.com cp /etc/resolv.conf /etc/resolv.conf.backup1 sed -i '/nameserver/d' /etc/resolv.conf echo "nameserver $MYMASTER_IP" >> /etc/resolv.conf sftp root@$MYMASTER:/var/lib/ipa/replica-info-$(hostname -s).testrelm.com.gpg \ /dev/shm ipa-replica-install -U --setup-ca --setup-dns \ --forwarder=192.168.122.1 -w Secret123 -p Secret123 \ /dev/shm/replica-info-$(hostname -s).testrelm.com.gpg
Created attachment 798635 [details] proposed patch Moves apr_signal after pipe creation, so when signal is handled, it has all the variables initialized properly in handler.
Scott can you retest with this scratch build of mod_wsgi? https://brewweb.devel.redhat.com/taskinfo?taskID=6296160
That does seemed to have fixed the issue from the manual and automated tests I ran today. I didn't see a crash from either.
*** Bug 1101652 has been marked as a duplicate of this bug. ***
I believe I am seeing this crash again with this version of httpd. Unfortunately, the automated test where I saw it no longer has the servers available to log in and grab more log files. httpd-2.2.15-38.el6.x86_64 Is this scheduled to be fixed in RHEL6.6? I'll also attach the latest abrt report we got on this one to confirm/deny that what I'm seeing is indeed the same. Thanks, Scott
Created attachment 925214 [details] abrt backtrace from rhel6.6 crash
*** Bug 1126987 has been marked as a duplicate of this bug. ***
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2014-1612.html