Bug 797426

Summary: Qpid AMQP daemon fails to load after latest updates
Product: Red Hat Enterprise Linux 6 Reporter: o.h.weiergraeber
Component: nssAssignee: Elio Maldonado Batiz <emaldona>
Status: CLOSED ERRATA QA Contact: Aleš Mareček <amarecek>
Severity: high Docs Contact:
Priority: high    
Version: 6.2CC: amarecek, jneedle, jross, jslagle, jwest, kengert, lzhaldyb, sgraf, skozina, syeghiay
Target Milestone: rcKeywords: Regression, ZStream
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: nss-3.13.1-4.el6 Doc Type: Bug Fix
Doc Text:
Cause: Some packages call nss without initializing as mandated by the API and nss did not protect itself against such improper usage. Consequence: This caused crashes on shutdown as some variables haven't been properly initialized. Such crashes were reported in the messaging daemon (qpidd) included in Red Hat Enterprise MRG after a recent update to nss. This occurred as qpidd made nss calls before initializing nss. Fix: Wit this fix nss now protects itself against potential improper use by client code. Result: With this update nss prevents qpidd, and other affected processes that may call nss without initializing as mandated by the API, from crashing
Story Points: ---
Clone Of:
: 797939 (view as bug list) Environment:
Last Closed: 2012-06-20 07:24:01 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 790746, 797939, 797959, 798461    
Attachments:
Description Flags
coredump of qpid crash
none
backtrace
none
threads
none
Also patch the NSS_Shutdown function rrelyea: review+

Description o.h.weiergraeber 2012-02-25 19:12:02 UTC
Description of problem:
After application of the latest updates today (nspr, nss, nss-util, nss-sysinit, xulrunner, iptables, dracut, dracut-kernel, selinux-policy, selinux-policy-targeted, firefox, yelp), I see this error message during boot:

-------------------------------------------------------------------------------
Starting Qpid AMQP daemon: bash: line 1:  2495 Segmentation fault (core dumped) /usr/sbin/qpidd --data-dir /var/lib/qpidd --daemon       [FAILED]
-------------------------------------------------------------------------------

No idea which component is actually faulty, maybe it's not qpid itself.


Version-Release number of selected component (if applicable):


How reproducible:
always


Steps to Reproduce:
1. apply latest updates (see above)
2. observe boot messages


Actual results:
error message during boot (see above)


Expected results:
no error message


Additional info:

Comment 3 Jeff Needle 2012-02-26 14:06:18 UTC
I received the same set of updates and did not have any issues.  Can you tell me more about your configuration (contents of /etc/qpidd.conf, sasl2 configurations, etc?) as well as any relevant snippets from /var/log/messages and qpidd.log?  Also, if you can post the core file, that would be helpful.

Comment 4 o.h.weiergraeber 2012-02-26 16:37:57 UTC
Created attachment 565908 [details]
coredump of qpid crash

Comment 5 o.h.weiergraeber 2012-02-26 16:47:15 UTC
OK, here we go.
I have a pretty much default installation of RHEL 6.2 Client and did not do any qpidd-related modifications.

This is my /etc/qpidd.conf:
--------------------------------------------------------------------------
cluster-mechanism=ANONYMOUS
auth=yes
--------------------------------------------------------------------------

In /etc/sasl2 there is another qpidd.conf (strange!):
--------------------------------------------------------------------------
pwcheck_method: auxprop
auxprop_plugin: sasldb
sasldb_path: /var/lib/qpidd/qpidd.sasldb

#following line stops spurious 'sql_select option missing' errors when
#cyrus-sql-sasl plugin is installed
sql_select: dummy select
--------------------------------------------------------------------------

and an smtpd.conf:
--------------------------------------------------------------------------
pwcheck_method: saslauthd
mech_list: plain login
--------------------------------------------------------------------------

'less messages | grep qpid' gives me:
--------------------------------------------------------------------------
Feb 26 16:51:07 osiris qpidd[2511]: 2012-02-26 16:51:07 notice Listening on TCP port 5672
Feb 26 16:51:07 osiris qpidd[2511]: 2012-02-26 16:51:07 notice SSL plugin not enabled, you must set --ssl-cert-db to enable it.
Feb 26 16:51:07 osiris qpidd[2511]: 2012-02-26 16:51:07 notice Broker running
Feb 26 16:51:07 osiris kernel: qpidd[2510]: segfault at 10 ip 00000035df0091c0 sp 00007fffe97f3cb8 error 4 in libpthread-2.12.so[35df000000+17000]
Feb 26 16:51:07 osiris abrt[2515]: saved core dump of pid 2510 (/usr/sbin/qpidd) to /var/spool/abrt/ccpp-2012-02-26-16:51:07-2510.new/coredump (1351680 bytes)
--------------------------------------------------------------------------

Interestingly, the directory ccpp-2012-02-26-16:51:07-2510.new does not exist, but three older ones do; maybe abrt does not allow more than three coredumps of the same kind???
I have attached the latest one stored.

Hope that helps to resolve the issue.

Comment 6 o.h.weiergraeber 2012-02-26 20:11:14 UTC
Just realized this bug is probably identical to
Bug 790746

Comment 7 Stanislav Graf 2012-02-27 07:54:53 UTC
Version-Release number of selected component (if applicable):
qpid-cpp-client-0.12-6.el6.x86_64
qpid-cpp-client-ssl-0.12-6.el6.x86_64
qpid-cpp-debuginfo-0.12-6.el6.x86_64
qpid-cpp-server-0.12-6.el6.x86_64
qpid-cpp-server-ssl-0.12-6.el6.x86_64

Comment 8 Stanislav Graf 2012-02-27 07:55:29 UTC
Created attachment 565976 [details]
backtrace

Comment 9 Stanislav Graf 2012-02-27 07:56:00 UTC
Created attachment 565977 [details]
threads

Comment 13 Elio Maldonado Batiz 2012-02-27 21:02:09 UTC
Created attachment 566141 [details]
Also patch the NSS_Shutdown function

These are the changes relative to the previous version:
--- nssbz784674-protect-against-calls-before-nss_init.patch	2012-02-27 12:56:46.084530509 -0800
+++ Bug-797426-protect-against-calls-before-nss_init.patch	2012-02-27 12:29:03.987826701 -0800
@@ -1,6 +1,8 @@
-diff -up mozilla/security/nss/lib/nss/nssinit.c.784672 mozilla/security/nss/lib/nss/nssinit.c
---- mozilla/security/nss/lib/nss/nssinit.c.784672	2012-01-26 14:43:46.232357231 -0800
-+++ mozilla/security/nss/lib/nss/nssinit.c	2012-01-26 14:50:55.830512565 -0800
+Index: mozilla/security/nss/lib/nss/nssinit.c
+===================================================================
+RCS file: /cvsroot/mozilla/security/nss/lib/nss/nssinit.c,v
+--- mozilla/security/nss/lib/nss/nssinit.c	18 Oct 2011 19:03:31 -0000	1.114
++++ /mozilla/security/nss/lib/nss/nssinit.c	25 Jan 2012 19:54:07 -0000
 @@ -944,6 +944,12 @@ NSS_RegisterShutdown(NSS_ShutdownFunc sF
  {
      int i;
@@ -14,7 +16,7 @@ diff -up mozilla/security/nss/lib/nss/ns
      PZ_Lock(nssInitLock);
      if (!NSS_IsInitialized()) {
  	PZ_Unlock(nssInitLock);
-@@ -1002,6 +1008,11 @@ NSS_UnregisterShutdown(NSS_ShutdownFunc
+@@ -1002,6 +1008,11 @@ NSS_UnregisterShutdown(NSS_ShutdownFunc 
  {
      int i;
  
@@ -26,7 +28,19 @@ diff -up mozilla/security/nss/lib/nss/ns
      PZ_Lock(nssInitLock);
      if (!NSS_IsInitialized()) {
  	PZ_Unlock(nssInitLock);
-@@ -1192,6 +1203,11 @@ NSS_ShutdownContext(NSSInitContext *cont
+@@ -1140,6 +1151,11 @@ SECStatus
+ NSS_Shutdown(void)
+ {
+     SECStatus rv;
++    /* make sure our lock and condition variable are initialized one and only
++     * one time */ 
++    if (PR_CallOnce(&nssInitOnce, nss_doLockInit) != PR_SUCCESS) {
++	return SECFailure;
++    }
+     PZ_Lock(nssInitLock);
+ 
+     if (!nssIsInitted) {
+@@ -1192,6 +1208,11 @@ NSS_ShutdownContext(NSSInitContext *cont
  {
      SECStatus rv = SECSuccess;

Comment 14 Bob Relyea 2012-02-27 21:12:30 UTC
Comment on attachment 566141 [details]
Also patch the NSS_Shutdown function

r+ rrelyea

Comment 15 Jeff Needle 2012-02-28 11:21:36 UTC
*** Bug 790746 has been marked as a duplicate of this bug. ***

Comment 18 Kai Engert (:kaie) (inactive account) 2012-02-28 17:34:10 UTC
*** Bug 797959 has been marked as a duplicate of this bug. ***

Comment 19 Suzanne Logcher 2012-02-28 18:01:18 UTC
*** Bug 797959 has been marked as a duplicate of this bug. ***

Comment 21 James Slagle 2012-02-29 14:22:04 UTC
*** Bug 798482 has been marked as a duplicate of this bug. ***

Comment 25 errata-xmlrpc 2012-06-20 07:24:01 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2012-0973.html