Bug 1102874

Summary:	RHEL6.5 sssd_be crash when debug_level 3 or higher
Product:	Red Hat Enterprise Linux 6	Reporter:	Scott Poore <spoore>
Component:	sssd	Assignee:	Jakub Hrozek <jhrozek>
Status:	CLOSED ERRATA	QA Contact:	Kaushik Banerjee <kbanerje>
Severity:	unspecified	Docs Contact:
Priority:	medium
Version:	6.5	CC:	grajaiya, jgalipea, lslebodn, mkosek, nsoman, pbrezina, preichl, sbose, spoore
Target Milestone:	rc	Keywords:	Reopened
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	sssd-1.11.6-1.el6	Doc Type:	Bug Fix
Doc Text:	No Documentation Needed	Story Points:	---
Clone Of:		Environment:
Last Closed:	2014-10-14 04:48:30 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Scott Poore 2014-05-29 18:09:06 UTC

Description of problem:

I'm seeing sssd_be crashing during sssd stop/starts.

Version-Release number of selected component (if applicable):
sssd-1.9.2-129.el6.s390x

How reproducible:
unknown but, seems very common.


Steps to Reproduce:
1.  Setup IPA server (my test used RHEL7)
2.  Setup IPA client on s390x
3.  service sssd stop; rm -rf /var/lib/sss/{db,mc}/*; service sssd start

Actual results:
sssd_be crashes 

Expected results:
no crash

Additional info:
will attach abrt shortly

Comment 3 Jakub Hrozek 2014-05-29 19:25:13 UTC

This could be a dupe of #1086264, if the crash is reproducable in your environment, could you try with the 6.6 packages?

Comment 4 Scott Poore 2014-05-30 15:01:24 UTC

Yes, I do believe that stopped the crashes.

Comment 5 Jakub Hrozek 2014-06-02 08:15:57 UTC

Upstream ticket:
https://fedorahosted.org/sssd/ticket/1006

Comment 6 Jakub Hrozek 2014-06-02 08:17:37 UTC

Closing as duplicate of #1086264 as per comment #4.

*** This bug has been marked as a duplicate of bug 1086264 ***

Comment 7 Scott Poore 2014-06-03 02:15:32 UTC

I'm not sure if it's relevant to declaring this a duplicate but, I just realized that the timing of the crash may not be during restart.

The crashes I am seeing in current attempted tests are happening during failed ssh logins.

Comment 8 Scott Poore 2014-06-03 03:08:52 UTC

Just a quick note, "sudo id" as a user also triggers a crash.

Comment 9 Scott Poore 2014-06-04 16:28:04 UTC

With Sumit's help, it was discovered that this crash is occurring when the debug_level is set to 3 or greater for the domain in sssd.conf.

================ fail with debug_level=10 =============

[root@ibm-z10-48 ~]# grep debug_level /etc/sssd/sssd.conf
debug_level = 10
debug_level = 10
debug_level = 10
debug_level = 10
debug_level = 10
debug_level = 10
debug_level = 10
debug_level = 10
[root@ibm-z10-48 ~]# date; ssh -l aduser1 $(hostname)
Wed Jun  4 10:01:09 EDT 2014
aduser1@ibm-z10-48.spoore06022245.test's password: 
Permission denied, please try again.
aduser1@ibm-z10-48.spoore06022245.test's password: 

[root@ibm-z10-48 ~]# tail /var/log/messages 
Jun  4 10:01:17 ibm-z10-48 abrtd: Directory 'ccpp-2014-06-04-10:01:17-20618' creation detected
Jun  4 10:01:17 ibm-z10-48 abrt[24188]: Saved core dump of pid 20618 (/usr/libexec/sssd/sssd_be) to /var/spool/abrt/ccpp-2014-06-04-10:01:17-20618 (35913728 bytes)
Jun  4 10:01:17 ibm-z10-48 sssd[be[spoore06022245.test]]: Starting up
Jun  4 10:01:24 ibm-z10-48 kernel: Program python tried to access /dev/mem between f0000->100000.
Jun  4 10:01:34 ibm-z10-48 abrtd: Sending an email...
Jun  4 10:01:35 ibm-z10-48 abrtd: Email was sent to: seceng-idm-qe-list
Jun  4 10:01:36 ibm-z10-48 abrtd: Duplicate: UUID
Jun  4 10:01:36 ibm-z10-48 abrtd: DUP_OF_DIR: /var/spool/abrt/ccpp-2014-06-03-13:18:19-5049
Jun  4 10:01:36 ibm-z10-48 abrtd: Deleting problem directory ccpp-2014-06-04-10:01:17-20618 (dup of ccpp-2014-06-03-13:18:19-5049)
Jun  4 10:01:36 ibm-z10-48 abrtd: No actions are found for event 'notify-dup'

================== works with debug_level <3 ===================

[root@ibm-z10-48 ~]# vim /etc/sssd/sssd.conf

[root@ibm-z10-48 ~]# grep -A2 "\[domain" /etc/sssd/sssd.conf
[domain/spoore06022245.test]
debug_level = 2

[root@ibm-z10-48 ~]# service sssd stop; rm -rf /var/lib/sss/{db,mc}/*; service sssd start
Stopping sssd: [  OK  ]
[  OK  ] sssd: [  OK  ]

[root@ibm-z10-48 ~]# date; ssh -l aduser1 $(hostname)
Wed Jun  4 10:07:45 EDT 2014
aduser1@ibm-z10-48.spoore06022245.test's password: 
...
-sh-4.1$ exit
logout
Connection to ibm-z10-48.spoore06022245.test closed.

[root@ibm-z10-48 ~]# tail /var/log/messages 
Jun  4 10:07:39 ibm-z10-48 sssd[ssh]: Shutting down
Jun  4 10:07:39 ibm-z10-48 sssd[pam]: Shutting down
Jun  4 10:07:39 ibm-z10-48 sssd[nss]: Shutting down
Jun  4 10:07:40 ibm-z10-48 sssd: Starting up
Jun  4 10:07:40 ibm-z10-48 sssd[be[spoore06022245.test]]: Starting up
Jun  4 10:07:40 ibm-z10-48 sssd[nss]: Starting up
Jun  4 10:07:40 ibm-z10-48 sssd[ssh]: Starting up
Jun  4 10:07:40 ibm-z10-48 sssd[pam]: Starting up
Jun  4 10:07:40 ibm-z10-48 sssd[sudo]: Starting up
Jun  4 10:07:40 ibm-z10-48 sssd[pac]: Starting up

================== example failing with debug_level=3 =====================

[root@ibm-z10-43 ~]# vim /etc/sssd/sssd.conf

[root@ibm-z10-43 ~]# service sssd stop; rm -rf /var/lib/sss/{db,mc}/*; service sssd start
Stopping sssd: [  OK  ]
[  OK  ] sssd: [  OK  ]

[root@ibm-z10-43 ~]# ssh -l aduser1 $(hostname)
aduser1@ibm-z10-43.spoore06022245.test's password: 
Permission denied, please try again.
aduser1@ibm-z10-43.spoore06022245.test's password: 

[root@ibm-z10-43 ~]# grep -A 2 "\[domain" /etc/sssd/sssd.conf
[domain/spoore06022245.test]
debug_level = 3

[root@ibm-z10-43 ~]# tail /var/log/messages 
Jun  4 09:48:58 ibm-z10-43 sssd[pam]: Starting up
Jun  4 09:48:58 ibm-z10-43 sssd[pac]: Starting up
Jun  4 09:48:58 ibm-z10-43 sssd[sudo]: Starting up
Jun  4 09:49:06 ibm-z10-43 abrt[9482]: Saved core dump of pid 9465 (/usr/libexec/sssd/sssd_be) to /var/spool/abrt/ccpp-2014-06-04-09:49:06-9465 (35762176 bytes)
Jun  4 09:49:06 ibm-z10-43 sssd[be[spoore06022245.test]]: Starting up
Jun  4 09:49:06 ibm-z10-43 abrtd: Directory 'ccpp-2014-06-04-09:49:06-9465' creation detected
Jun  4 09:49:15 ibm-z10-43 kernel: Program python tried to access /dev/mem between f0000->100000.
Jun  4 09:50:08 ibm-z10-43 abrtd: Sending an email...
Jun  4 09:50:09 ibm-z10-43 abrtd: Email was sent to: seceng-idm-qe-list
Jun  4 09:50:09 ibm-z10-43 abrtd: New problem directory /var/spool/abrt/ccpp-2014-06-04-09:49:06-9465, processing

Comment 18 Jakub Hrozek 2014-06-17 17:15:29 UTC

Upstream ticket:
https://fedorahosted.org/sssd/ticket/2362

Comment 19 Jakub Hrozek 2014-06-17 17:19:39 UTC

The offending DEBUG message was fixed during the rewrite of nested LDAP group processing.

Scott, given the DEBUG message that caused the crash is fixed in 6.6 and 7.0 would you agree with including this bug in the 6.6 errata?

I would prefer not to close this ticket because for users running 6.5 who might hit this issue it would be best if they could search the list of open bugzillas and discover it's a known problem.

Comment 20 Scott Poore 2014-06-17 17:34:43 UTC

I'm good with it being included in 6.6.  That's a very good reason to include it in my opinion.

Comment 23 Scott Poore 2014-07-16 00:49:28 UTC

Verified.

Version ::

Results ::

First confirm failure:

[root@ibm-z10-29 ~]# vi /etc/sssd/sssd.conf
[root@ibm-z10-29 ~]# service sssd stop; rm -rf /var/lib/sss/{db,mc}/*; service sssd startStopping sssd: [  OK  ]
[  OK  ] sssd: [  OK  ]
[root@ibm-z10-29 ~]# ssh -l "aduser1" ibm-z10-29.spoore07151535.test "echo 'login successful'"
aduser1@ibm-z10-29.spoore07151535.test's password: 
Permission denied, please try again.
aduser1@ibm-z10-29.spoore07151535.test's password: 


Then upgrade:

[root@ibm-z10-29 yum.local.d]# yum update sssd
...           

Updated:
  sssd.s390x 0:1.11.6-3.el6

Dependency Updated:
  libipa_hbac.s390x 0:1.11.6-3.el6                libipa_hbac-python.s390x 0:1.11.6-3.el6              
  libsss_idmap.s390x 0:1.11.6-3.el6               sssd-client.s390x 0:1.11.6-3.el6                     

Replaced:
  libsss_autofs.s390x 0:1.9.2-129.el6                 libsss_sudo.s390x 0:1.9.2-129.el6                

Complete!

Test again:

[root@ibm-z10-29 yum.local.d]# grep '\[domain' -A1 /etc/sssd/sssd.conf 
[domain/spoore07151535.test]
debug_level = 10
[root@ibm-z10-29 yum.local.d]# service sssd stop; rm -rf /var/lib/sss/{db,mc}/*; service sssd start
Stopping sssd: [  OK  ]
Starting sssd: [  OK  ]
[root@ibm-z10-29 yum.local.d]# ssh -l aduser1 ibm-z10-29.spoore07151535.test "echo 'login successful'"
aduser1@ibm-z10-29.spoore07151535.test's password: 
login successful
[root@ibm-z10-29 yum.local.d]#

Comment 24 errata-xmlrpc 2014-10-14 04:48:30 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1375.html