Bug 1261186 - sssd_be general protection in libsss_idmap.so.0.4.0. [NEEDINFO]
sssd_be general protection in libsss_idmap.so.0.4.0.
Status: CLOSED WORKSFORME
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: sssd (Show other bugs)
7.1
x86_64 Linux
unspecified Severity high
: rc
: ---
Assigned To: Jakub Hrozek
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-09-08 15:59 EDT by Todd H. Poole
Modified: 2016-01-07 11:38 EST (History)
13 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-01-07 11:38:27 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
jhrozek: needinfo? (toddhpoolework)


Attachments (Terms of Use)

  None (edit)
Description Todd H. Poole 2015-09-08 15:59:34 EDT
Description of problem:
kernel: traps: sssd_be general protection in libsss_idmap.so.0.4.0.

After several days of light use (30 or so users logging in and out several times a day), sssd_be begins misbehaving with the following messages showing up in /var/log/messages immediately after each login attempt:

Sep  8 10:59:48 redacted systemd: Starting Session 698 of user toddhpoole.
Sep  8 10:59:48 redacted systemd: Started Session 698 of user toddhpoole.
Sep  8 10:59:48 redacted systemd-logind: New session 698 of user toddhpoole.
Sep  8 10:59:48 redacted systemd-logind: Removed session 698.
Sep  8 10:59:48 redacted kernel: traps: sssd_be[32615] general protection ip:7f97fd56d331 sp:7fff0906baf0 error:0 in libsss_idmap.so.0.4.0[7f97fd56a000+5000]

This prevents users from successfully logging in.

Neither restarting the sssd service (via 'systemctl restart sssd') nor purging the sss cache (via 'sss_cache -E') and then trying to reconnect via SSH appear to resolve the issue:

Sep  8 11:15:35 redacted systemd: Started System Security Services Daemon.
Sep  8 11:15:51 redacted kernel: traps: sssd_be[792] general protection ip:7fac644d2331 sp:7fff195e5c30 error:0 in libsss_idmap.so.0.4.0[7fac644cf000+5000]
Sep  8 11:15:51 redacted sssd[be[redacted.domain.local]]: Starting up
Sep  8 11:17:17 redacted kernel: traps: sssd_be[800] general protection ip:7ffa2b176331 sp:7fff60e65370 error:0 in libsss_idmap.so.0.4.0[7ffa2b173000+5000]
Sep  8 11:17:17 redacted sssd[be[redacted.domain.local]]: Starting up
Sep  8 11:17:53 redacted kernel: traps: sssd_be[814] general protection ip:7fb30dab4331 sp:7fff9038f100 error:0 in libsss_idmap.so.0.4.0[7fb30dab1000+5000]
Sep  8 11:17:53 redacted sssd[be[redacted.domain.local]]: Starting up
Sep  8 11:19:06 redacted kernel: traps: sssd_be[822] general protection ip:7f4030cc0331 sp:7fff4a949aa0 error:0 in libsss_idmap.so.0.4.0[7f4030cbd000+5000]

Deleting the cache database file (via 'rm -f /var/lib/sss/db/cache_DOMAIN_redacted.ldb') and then restarting sssd (via 'systemctl restart sssd') does appear to resolve the issue.

Version-Release number of selected component (if applicable):
[root@redacted ~]# sssd --version
1.12.2
[root@redacted ~]# rpm -qa | grep "sssd"
sssd-ldap-1.12.2-58.el7_1.6.x86_64
sssd-common-pac-1.12.2-58.el7_1.6.x86_64
sssd-ad-1.12.2-58.el7_1.6.x86_64
python-sssdconfig-1.12.2-58.el7_1.6.noarch
sssd-krb5-common-1.12.2-58.el7_1.6.x86_64
sssd-krb5-1.12.2-58.el7_1.6.x86_64
sssd-ipa-1.12.2-58.el7_1.6.x86_64
sssd-1.12.2-58.el7_1.6.x86_64
sssd-debuginfo-1.12.2-58.el7_1.6.x86_64
sssd-client-1.12.2-58.el7_1.6.x86_64
sssd-common-1.12.2-58.el7_1.6.x86_64
sssd-proxy-1.12.2-58.el7_1.6.x86_64

How reproducible:
The issue presents itself sporadically. In most cases, after about 6 or 7 days of uptime, but, I've witnessed failures starting to occur as early as 12 hours and as late as 16 days. Deleting the cache database file then restarting sssd appears to reset the timer.

Steps to Reproduce:
1. Wait several days with a small number of users logging in and out of the system several times a day.
2. Observe after several days sssd_be begins failing, preventing users from logging in.

Actual results:
sssd_be fails.

Expected results:
sssd_be does not fail.

Additional info:
Google yields a surprisingly small number of hits for this failure. I see there was a similar bug posted last year (https://bugzilla.redhat.com/show_bug.cgi?id=1079237), but that was purportedly fixed by the version I have. There's also this posting (http://freeipa-users.redhat.narkive.com/0Hl9a7R4/sssd-sssd-be-crashing-on-rhel-6-2) which references the same general protection fault, but that user has a significantly different environment than we do. Core dumps are not available for this machine, and /var/log/sssd/sssd.log is 0B in size (as is /var/log/sssd/sssd_redacted.domain.local). I'd be happy to install abrt on this machine to capture core dumps if this bug is a true bug and not a DUPLICATE of a pre-existing one that I might have missed or provide additional logs with sensitive information redacted if requested.

Thank you,
Todd H. Poole
Comment 1 Lukas Slebodnik 2015-09-09 04:21:29 EDT
(In reply to Todd H. Poole from comment #0)
> Description of problem:
> kernel: traps: sssd_be general protection in libsss_idmap.so.0.4.0.
> 
> After several days of light use (30 or so users logging in and out several
> times a day), sssd_be begins misbehaving with the following messages showing
> up in /var/log/messages immediately after each login attempt:
> 
> Sep  8 10:59:48 redacted systemd: Starting Session 698 of user toddhpoole.
> Sep  8 10:59:48 redacted systemd: Started Session 698 of user toddhpoole.
> Sep  8 10:59:48 redacted systemd-logind: New session 698 of user toddhpoole.
> Sep  8 10:59:48 redacted systemd-logind: Removed session 698.
> Sep  8 10:59:48 redacted kernel: traps: sssd_be[32615] general protection
> ip:7f97fd56d331 sp:7fff0906baf0 error:0 in
> libsss_idmap.so.0.4.0[7f97fd56a000+5000]
> 
> This prevents users from successfully logging in.
> 
> Neither restarting the sssd service (via 'systemctl restart sssd') nor
> purging the sss cache (via 'sss_cache -E') and then trying to reconnect via
> SSH appear to resolve the issue:
> 
> Sep  8 11:15:35 redacted systemd: Started System Security Services Daemon.
> Sep  8 11:15:51 redacted kernel: traps: sssd_be[792] general protection
> ip:7fac644d2331 sp:7fff195e5c30 error:0 in
> libsss_idmap.so.0.4.0[7fac644cf000+5000]
> Sep  8 11:15:51 redacted sssd[be[redacted.domain.local]]: Starting up
> Sep  8 11:17:17 redacted kernel: traps: sssd_be[800] general protection
> ip:7ffa2b176331 sp:7fff60e65370 error:0 in
> libsss_idmap.so.0.4.0[7ffa2b173000+5000]
> Sep  8 11:17:17 redacted sssd[be[redacted.domain.local]]: Starting up
> Sep  8 11:17:53 redacted kernel: traps: sssd_be[814] general protection
> ip:7fb30dab4331 sp:7fff9038f100 error:0 in
> libsss_idmap.so.0.4.0[7fb30dab1000+5000]
> Sep  8 11:17:53 redacted sssd[be[redacted.domain.local]]: Starting up
> Sep  8 11:19:06 redacted kernel: traps: sssd_be[822] general protection
> ip:7f4030cc0331 sp:7fff4a949aa0 error:0 in
> libsss_idmap.so.0.4.0[7f4030cbd000+5000]
> 
> Deleting the cache database file (via 'rm -f
> /var/lib/sss/db/cache_DOMAIN_redacted.ldb') and then restarting sssd (via
> 'systemctl restart sssd') does appear to resolve the issue.
> 
> Version-Release number of selected component (if applicable):
> [root@redacted ~]# sssd --version
> 1.12.2
> [root@redacted ~]# rpm -qa | grep "sssd"
> sssd-ldap-1.12.2-58.el7_1.6.x86_64
> sssd-common-pac-1.12.2-58.el7_1.6.x86_64
> sssd-ad-1.12.2-58.el7_1.6.x86_64
> python-sssdconfig-1.12.2-58.el7_1.6.noarch
> sssd-krb5-common-1.12.2-58.el7_1.6.x86_64
> sssd-krb5-1.12.2-58.el7_1.6.x86_64
> sssd-ipa-1.12.2-58.el7_1.6.x86_64
> sssd-1.12.2-58.el7_1.6.x86_64
> sssd-debuginfo-1.12.2-58.el7_1.6.x86_64
> sssd-client-1.12.2-58.el7_1.6.x86_64
> sssd-common-1.12.2-58.el7_1.6.x86_64
> sssd-proxy-1.12.2-58.el7_1.6.x86_64
> 

This BZ is filed against fedora 22. Which has sssd-1.12.5.
So please try to test with this version.

If your plan was to failed against el7.1 then It would be better to use
different component in Bugzilla. BTW There were some updates in el7.
You might try to reproduce with 1.12.2-58.el7_1.14.

If there will be still a problem then you might want to test with backported
version from fedora 22.
https://copr.fedoraproject.org/coprs/lslebodn/sssd-1-12/
Comment 2 Todd H. Poole 2015-09-09 18:55:11 EDT
Thank you for the feedback Lukas, I'll give those a shot.

Unfortunately, given the relatively infrequent nature of these failures, it'll be several days (if not weeks) before I'm able to report back again. Expect an update no later than the 24th of September.

For the sake of correctness, I've also gone ahead and updated this report's components to more accurately reflect the actual environment and installed packages.
Comment 4 Jakub Hrozek 2015-09-10 03:57:10 EDT
Thank you for changing the bug version.

I'll set the needinfo flag to make it clear we need the corefile or logs (ideally with a totally up-to-date version) to help with the issue.
Comment 5 Todd H. Poole 2015-09-24 17:54:13 EDT
It's been 15 days since the latest patch/bugfix release was applied to one of our test/staging clusters, and I'm pleased to report we've not seen the issue return.

We've begun deploying these changes to our production environment, which will have a significantly higher number of users exercising these services. If these failures do not return after a few weeks of heavy load in our production environment, then I think we can consider this issue resolved.

I'll report back if anything changes. Thanks gentlemen.
Comment 6 Jakub Hrozek 2015-09-25 04:13:51 EDT
(In reply to Todd H. Poole from comment #5)
> It's been 15 days since the latest patch/bugfix release was applied to one
> of our test/staging clusters, and I'm pleased to report we've not seen the
> issue return.
> 
> We've begun deploying these changes to our production environment, which
> will have a significantly higher number of users exercising these services.
> If these failures do not return after a few weeks of heavy load in our
> production environment, then I think we can consider this issue resolved.
> 
> I'll report back if anything changes. Thanks gentlemen.

Thank you very much for coming back.

Please just close this bugzilla if the issue doesn't hit you in the production environment.
Comment 7 Jakub Hrozek 2016-01-07 11:38:27 EST
We haven't heard on this issue for quite some time, closing.

Note You need to log in before you can comment on or make changes to this bug.