Bug 972699

Hi Peter,

Do you have a corefile for the segfault? It would help us a lot.
If you're worried about the content of the corefile, you could check it with the "strings" utility first for any private data.

To get the core file or the backtrace, you can simply install and start abrt, it should get the crash data automatically.

As per the error where the sssd doesn't seemingly segfault, can you attach the logs from the period when the problem happened? Maybe we could at least see what was going on. I'm concerned about the SSSD not reconnecting after a problem, that seems quite bad.

Unfortunately, we do not have the core files from the earlier segfaults. If a segfault happens on the new system, I will give it to you. 

Will attach the logs.

Created attachment 759442 [details]
sssd_default.log-20130609.gz

Created attachment 759443 [details]
sssd.log-20130609.gz

Created attachment 759444 [details]
sssd_nss.log-20130609.gz

Created attachment 759445 [details]
sssd_pam.log-20130609.gz

Hi,
I'm sorry for late answer. Did you manage to get the core file? It would help a lot. sssd_pam process is terminated abruptly so there should be a core file generated by abrt (if enabled).

I see one scenario twice in the logs. SSSD gets requests for quite a lot users, probably a /home directory is being listed? This enumeration is interrupted by ssh login of user "cenddrif", returning success to sssd_pam. Does this user contain some special values in LDAP? Are you able to login with him?

Also, can you test if setting 'client_idle_timeout = 600' in you sssd.conf domain section [domain/default] helps?

Thank you.

As I mentioned in comment #4, we do not have the the core files from the earlier segfaults. This server is no longer part of our organization. 

"cenddrif" is a pretty normal account, used by the Nagios checks. The login works fine. 

Strangely enough, we haven't seen this problem for a while. I've added 'client_idle_timeout = 600' anyway.

I'll set the needinfo flag again to make it clear we need the core file in order to provide a fix and leave the bug open for a while longer.

sssd on usdemo.thinlinc.com died again this weekend. It's difficult to tell if it actually crashed or not - no traces found in dmesg about any segfault. 

Historically, we have had some problems with missing core dumps when ABRT is installed. I would love to uninstall ABRT, but unfortunately this is not possible because sabayon (which we use) depends on it. But I have at least stopped the abrtd service. 

Also, I've added:

DAEMON_COREFILE_LIMIT=999999

...to /etc/init.d/sssd. I've verified that "kill -TRAP" on the sssd process generates a core file in /, so hopefully, we will get a core file the next time it crashes. If it *is* a segfault...

(In reply to Peter Åstrand from comment #12)
> sssd on usdemo.thinlinc.com died again this weekend. It's difficult to tell
> if it actually crashed or not - no traces found in dmesg about any segfault. 
> 

Anything interesting in the sssd log files? Fatal errors would have been printed even without explicit debug_level setting.

> Historically, we have had some problems with missing core dumps when ABRT is
> installed. I would love to uninstall ABRT, but unfortunately this is not
> possible because sabayon (which we use) depends on it. But I have at least
> stopped the abrtd service. 
> 
> Also, I've added:
> 
> DAEMON_COREFILE_LIMIT=999999
> 
> ...to /etc/init.d/sssd. I've verified that "kill -TRAP" on the sssd process
> generates a core file in /, so hopefully, we will get a core file the next
> time it crashes. If it *is* a segfault...

OK, sounds good, thank you for following up and for the patience.

Hi Peter,

I think this bug is actually the same as https://bugzilla.redhat.com/show_bug.cgi?id=984814 for which we have a candidate patch for. If you're interested and still seeing the problem, we can build you test packages..

Created attachment 785594 [details]
Core was generated by `/usr/libexec/sssd/sssd_pam --debug-to-files'.

sssd died last friday again. Attaching core.

(In reply to Jakub Hrozek from comment #14)
> Hi Peter,
> 
> I think this bug is actually the same as
> https://bugzilla.redhat.com/show_bug.cgi?id=984814 for which we have a

"You are not authorized to access bug #984814."

It is clear, that there was two segfaults (https://bugzilla.redhat.com/show_bug.cgi?id=972699#c0). The first in sssd_pam and the second in sssd_nss

>May  6 21:23:35 lola kernel: sssd_pam[1988]: segfault at 10 ip 0000000000410fc0 >sp 00007fff60221660 error 4 in sssd_pam (deleted)[400000+75000]
>May  6 21:23:35 lola sssd[pam]: Starting up
>May  6 21:23:37 lola sssd[pam]: Starting up
>May  6 21:23:41 lola sssd[pam]: Starting up
>May  6 21:23:41 lola sssd[be[default]]: Shutting down
>May  6 21:23:41 lola kernel: sssd_nss[1987]: segfault at c4 ip 0000003252024fbc >sp 00007ffff9ab8620 error 4 in libdbus-1.so.3.4.0[3252000000+40000]

Jakub thought about the second crash as a duplicate of bug #984814 (private bug)
Upstream ticket for that bug is located here: https://fedorahosted.org/sssd/ticket/2018
We cannot be sure without coredump whether it is the same problem.

The first crash looked very similar to upstream ticket https://fedorahosted.org/sssd/ticket/1869. After deep analysis of coredump, we found out that it is different bug. Cause of crash is dereference of NULL pointer.

Thank you very much for coredump. It was very helpful, but I need to discuss this problem with more experienced sssd developers. I am not really sure how it could happen that pointer is NULL.

sssd crashed again for us a few hours ago. I can see that both upstream bugs are now fixed, so what about releasing an updated RPM package?

Hi,

There are two issues. One is a crash in the memory cache code which has been fixed and will be released as an update no later than 6.5 (perhaps sooner, details tbd). I can give you a 6.5 candidate built for 6.4 now if you like, testing is always very much welcome.

The other issue is the NULL dereference in the PAM responder. Unfortunately the coredump tells us where the SSSD crashed but not how we got there. But you're right we need to have some kind of resolution here. I will revive the discussion Lukas started on the upstream mailing list and suggest that we at least add a NULL check and several additional DEBUG statements that would prevent the crash in the future and give us a better hint on what happened.

By the way given that you are hit by this issue in a production environment, I suspect that running the pam responder under valgrind would not be a viable option? There would be some performance penalty, but functionality should be retained.

(In reply to Jakub Hrozek from comment #20)
> Hi,
> 
> There are two issues. One is a crash in the memory cache code which has been
> fixed and will be released as an update no later than 6.5 (perhaps sooner,
> details tbd). I can give you a 6.5 candidate built for 6.4 now if you like,
> testing is always very much welcome.

Since it only crashes once a month or so right now, I think we can wait. 


>By the way given that you are hit by this issue in a production environment, I >suspect that running the pam responder under valgrind would not be a viable >option? There would be some performance penalty, but functionality should be >retained.

I'd like to avoid this if possible. It's a remote system.

Could you also attach last coredump? I think it can confirm our assumption
about dereference of NULL pointer and that it is not the same problem like
https://fedorahosted.org/sssd/ticket/1869

Created attachment 793556 [details]
core.2013-09-03T15:03.gz

I actually got two core files within the hour, attaching both.

Created attachment 793562 [details]
core.2013-09-03T15:43.gz

Thank you very much for these coredumps.
I analysed them and it is the same problem as https://fedorahosted.org/sssd/ticket/1869 (which has already been fixed).
Jakub can provide test package for you.

Hi Peter,

you can download the test builds here:
http://jhrozek.fedorapeople.org/sssd-test-builds/sssd-pam-crash-rhbz972699/

It's the latest 6.4.z packages plus two fixes - one that would fix the crashes in the memcache and one that would prevent the pam responder from crashing on timeout.

I'm going to close this bug as duplicate of #951086 as Lukas' investigation shows they are the same. Feel free to use the test packages in comment #27.

Thank  you for reporting the issue and working with us towards fixing it!

*** This bug has been marked as a duplicate of bug 951086 ***

(In reply to Jakub Hrozek from comment #28)
> I'm going to close this bug as duplicate of #951086 as Lukas' investigation
> shows they are the same. Feel free to use the test packages in comment #27.
> 
> Thank  you for reporting the issue and working with us towards fixing it!
> 
> *** This bug has been marked as a duplicate of bug 951086 ***

I think this is totally wrong, since bug 951086 is private. It is certainly very impractical, since it will be, for example, impossible of following the progress of how and when updated RPMs are available.

IMO, DUPLICATE should be used for cases when two users accidently reported the same bug twice, and when a public user can easily abandon their own bug in favour of another. In some rating systems, you will also actually get a penalty when reporting a bug which was later closed as DUPLICATE, because the reporter did not try hard enough of finding a suitable existing report before creating another one. I don't want to look like a fool just because you have some internal information that you don't want to reveal. (Also, I don't understand why you do not use the built in access features of Bugzilla, which makes it possible to restrict comments/attachements/etc, so that you can use one single bug.)

Would you please reconsider this DUPLICATE decision? Why not close the internal report as DUPLICATE instead?

(In reply to Peter Åstrand from comment #29)
> (In reply to Jakub Hrozek from comment #28)
> > I'm going to close this bug as duplicate of #951086 as Lukas' investigation
> > shows they are the same. Feel free to use the test packages in comment #27.
> > 
> > Thank  you for reporting the issue and working with us towards fixing it!
> > 
> > *** This bug has been marked as a duplicate of bug 951086 ***
> 

I'm very sorry about the negative experience you had, Peter. To be honest, at first, I didn't realize the other bug was marked as private.. Let me try to explain what's going on:

> I think this is totally wrong, since bug 951086 is private. It is certainly
> very impractical, since it will be, for example, impossible of following the
> progress of how and when updated RPMs are available.
> 

You're completely right. However, that bug came through our support systems and for various reasons, mostly because content copied from the support system might include sensitive data, such bugs are marked private as default.

> IMO, DUPLICATE should be used for cases when two users accidently reported
> the same bug twice, and when a public user can easily abandon their own bug
> in favour of another. 

I agree.

> In some rating systems, you will also actually get a
> penalty when reporting a bug which was later closed as DUPLICATE, because
> the reporter did not try hard enough of finding a suitable existing report
> before creating another one. I don't want to look like a fool just because
> you have some internal information that you don't want to reveal.

I assure you that there is no penalty and I value the fact that you took the time to report the bug and worked with us towards confirming it is a known issue very much. Actually, I don't think there was any other way because as you noted, you couldn't have an idea about the other bug. Also, the problem you hit was not a trivial one and required careful investigation of the core file.

>  (Also, I
> don't understand why you do not use the built in access features of
> Bugzilla, which makes it possible to restrict comments/attachements/etc, so
> that you can use one single bug.)
> 

I would like to make the bugzilla public and only mark certain comments as private as you suggested, but I need to check if our process allows that. I'm sorry about the inconvenience before this issue is sorted out.

> Would you please reconsider this DUPLICATE decision? Why not close the
> internal report as DUPLICATE instead?

Because that bug was already approved for inclusion in the next RHEL release and by marking that one as duplicate of this one the approval would have to be redone.

I marked bug 951086 as public. You should be able to see it now.

(In reply to Jakub Hrozek from comment #31)
> I marked bug 951086 as public. You should be able to see it now.

Thanks.