Bug 1079237

Summary: sssd is crashing after several quick invokes of automount -m
Product: [Fedora] Fedora Reporter: habicht
Component: sssdAssignee: Jakub Hrozek <jhrozek>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 20CC: abokovoy, jhrozek, lslebodn, pbrezina, sbose, sgallagh, ssorce
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: sssd-1.11.5.1-1.fc20 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-04-24 07:36:19 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
core generated with crashing automount/sssd
none
sssd log files
none
core files from abrtd none

Description habicht 2014-03-21 09:23:36 UTC
Description of problem: 

I have a strange problem with sssd, ldap and autofs. I am using sssd and ldap for authentication and automount. Our fileserver has 4 shares, while the 4. is a new one. The mounting of the 4. share is not stabil and the systems have problems to see the possible mounts. While still investigating this problem i found a strange effect now describing here:

When i involve automount -m the output is sometimes hanging at this strange share. But not everytime. When i re-involve automount -m very quick and very often and it hangs again and than make several CTRL-C, than sssd is crashed.


Version-Release number of selected component (if applicable):
sssd-common-1.11.4-1.fc20.x86_64
sssd-client-1.11.4-1.fc20.i686
sssd-1.11.4-1.fc20.x86_64
sssd-ldap-1.11.4-1.fc20.x86_64
sssd-client-1.11.4-1.fc20.x86_64
autofs-5.0.7-40.fc20.x86_64

How reproducible:

See above


Steps to Reproduce:
1. See above
2.
3.

Actual results:

Making several CTRL-C at hanging automount -m output:

....
Mount point: /data

source(s):
Segmentation fault (core dumped)

And than:

[root@pxe-122 log]# systemctl status sssd
sssd.service - System Security Services Daemon
   Loaded: loaded (/usr/lib/systemd/system/sssd.service; enabled)
   Active: failed (Result: exit-code) since Fri 2014-03-21 09:28:09 CET; 5s ago
  Process: 488 ExecStart=/usr/sbin/sssd -D -f (code=exited, status=0/SUCCESS)
 Main PID: 563 (code=exited, status=1/FAILURE)




Expected results:

Normal output of automount -m . No crash of sssd.


Additional info:

Mar 21 09:27:18 pxe-122 kernel: [ 1571.044469] traps: sssd_be[1164] general protection ip:7fe295f1561a sp:7fffcbb365f8 error:0 in libc-2.18.so[7fe295e8f000+1b4000]
Mar 21 09:27:18 pxe-122 kernel: traps: sssd_be[1164] general protection ip:7fe295f1561a sp:7fffcbb365f8 error:0 in libc-2.18.so[7fe295e8f000+1b4000]
Mar 21 09:27:18 pxe-122 sssd[be[default]]: Starting up
Mar 21 09:27:19 pxe-122 automount[4745]: setautomntent: lookup(sss): setautomntent: No such file or directory
Mar 21 09:27:21 pxe-122 kernel: [ 1574.493951] traps: sssd_be[4751] general protection ip:7f8b58e2f61a sp:7fffd0bf44a8 error:0 in libc-2.18.so[7f8b58da9000+1b4000]
Mar 21 09:27:21 pxe-122 kernel: traps: sssd_be[4751] general protection ip:7f8b58e2f61a sp:7fffd0bf44a8 error:0 in libc-2.18.so[7f8b58da9000+1b4000]
Mar 21 09:27:23 pxe-122 sssd[be[default]]: Starting up
Mar 21 09:27:47 pxe-122 kernel: [ 1600.703958] traps: sssd_be[4764] general protection ip:7f567182761a sp:7fff371c6ed8 error:0 in libc-2.18.so[7f56717a1000+1b4000]
Mar 21 09:27:47 pxe-122 kernel: traps: sssd_be[4764] general protection ip:7f567182761a sp:7fff371c6ed8 error:0 in libc-2.18.so[7f56717a1000+1b4000]
Mar 21 09:27:51 pxe-122 sssd[be[default]]: Starting up
Mar 21 09:28:09 pxe-122 kernel: [ 1622.336917] traps: sssd_be[4834] general protection ip:7fcba70b861a sp:7fff04291f88 error:0 in libc-2.18.so[7fcba7032000+1b4000]
Mar 21 09:28:09 pxe-122 kernel: traps: sssd_be[4834] general protection ip:7fcba70b861a sp:7fff04291f88 error:0 in libc-2.18.so[7fcba7032000+1b4000]
Mar 21 09:28:09 pxe-122 kernel: [ 1622.338391] automount[4840]: segfault at 0 ip 00007fd58451791f sp 00007fffa34b86b0 error 4 in libsss_autofs.so[7fd584515000+4000]
Mar 21 09:28:09 pxe-122 sssd[autofs]: Shutting down
Mar 21 09:28:09 pxe-122 kernel: automount[4840]: segfault at 0 ip 00007fd58451791f sp 00007fffa34b86b0 error 4 in libsss_autofs.so[7fd584515000+4000]
Mar 21 09:28:09 pxe-122 sssd[pam]: Shutting down
Mar 21 09:28:09 pxe-122 sssd[nss]: Shutting down
Mar 21 09:28:09 pxe-122 systemd: sssd.service: main process exited, code=exited, status=1/FAILURE
Mar 21 09:28:09 pxe-122 systemd: Unit sssd.service entered failed state.

Comment 1 Lukas Slebodnik 2014-03-21 09:35:20 UTC
Could you provide coredump?

If you have installed abrt it should be in directory /var/tmp/abrt/ccpp-<date>

Comment 2 habicht 2014-03-21 10:02:15 UTC
Created attachment 877190 [details]
core generated with crashing automount/sssd

I have only this in the moment. But this is not from /var/tmp/abrt/. When you need a different one i can try to crash a system again.

Comment 3 Jakub Hrozek 2014-03-21 11:05:53 UTC
Upstream ticket:
https://fedorahosted.org/sssd/ticket/2288

Comment 4 Lukas Slebodnik 2014-03-21 11:59:23 UTC
I would say there are two different issues(crashes):

1. automount used sss_autofs plugin for retrieving maps
2. sss_autofs plugin sent request to sssd_be proccess.
3. sssd_be should download automounter maps from from ldap server.

Mar 21 09:28:09 pxe-122 kernel: traps: sssd_be[4834] general protection ip:7fcba70b861a sp:7fff04291f88 error:0 in libc-2.18.so[7fcba7032000+1b4000]
^^^^^^^
There should be coredump from sssd_be

4. sssd_be didn't replied to sss_autofs plugin.
5. sss_autofs plugin didn't handle failed requesst properly.

Mar 21 09:28:09 pxe-122 kernel: [ 1622.338391] automount[4840]: segfault at 0 ip 00007fd58451791f sp 00007fffa34b86b0 error 4 in libsss_autofs.so[7fd584515000+4000]
^^^^^^^^
This is a coredump you provided in attachment 877190 [details]

Could you also provide coredump from sssd_be?
The lod file from sssd_be process will be helpful as well.
You will need to add debug_level=10 into "domain" and "autofs" section and send log files from directory /var/log/sssd. You can send them directly to me.

Comment 5 habicht 2014-03-21 13:34:41 UTC
Created attachment 877271 [details]
sssd log files

This are sssd log files debug level 10.

I am not able to find a core for sssd_be. 
abrtd is running. Any idea how i can create a core?

Comment 6 habicht 2014-03-21 14:43:26 UTC
Created attachment 877321 [details]
core files from abrtd

core files from abrtd

Comment 7 Lukas Slebodnik 2014-03-21 15:53:23 UTC
Thank you very much for coredump and log files.

sssd_be crashed due to use after free error.

I can see in log files some LDAP entries have more then one attribute "automountKey"

[sdap_process_message] (0x4000): Message type: [LDAP_RES_SEARCH_ENTRY]
[sdap_parse_entry] (0x4000): OriginalDN: [cn=teamviewer9,nisMapName=auto.opt,ou=ea,dc=ims,dc=uni-hannover,dc=de].
[sdap_parse_range] (0x2000): No sub-attributes for [cn]
[sdap_parse_range] (0x2000): No sub-attributes for [nisMapEntry]
[sdap_parse_range] (0x2000): No sub-attributes for [objectClass]
[sdap_process_result] (0x2000): Trace: sh[0x7fdaf2bf4e10], connected[1], ops[0x7fdaf2c07360], ldap[0x7fdaf2bfcc20]
[sdap_process_result] (0x2000): Trace: ldap_result found nothing!
[sdap_process_result] (0x2000): Trace: sh[0x7fdaf2bf4e10], connected[1], ops[0x7fdaf2c07360], ldap[0x7fdaf2bfcc20]
[sdap_process_message] (0x4000): Message type: [LDAP_RES_SEARCH_RESULT]
[sdap_get_generic_ext_done] (0x0400): Search result: Success(0), no errmsg set
[automntmaps_process_members_done] (0x2000): No more search bases to try
[automntmaps_process_members_done] (0x0400): Search for autofs entries, returned 46 results.
[sdap_get_automntmap_done] (0x0400): automount map members received
[sdap_autofs_setautomntent_save] (0x1000): Got 46 map entries from LDAP
[get_autofs_entry_attr] (0x0020): Expected one entry got 2
[sdap_autofs_setautomntent_save] (0x0080): Malformed entry, skipping

If you fix following ldap entries in LDAP server sssd_autofs should work
cn=teamviewer9,nisMapName=auto.opt,ou=ea,dc=ims,dc=uni-hannover,dc=de
cn=envmodules,nisMapName=auto.appl,ou=ea,dc=ims,dc=uni-hannover,dc=de
cn=eod,nisMapName=auto.appl,ou=ea,dc=ims,dc=uni-hannover,dc=de

Thank you very much for report.

Comment 8 Lukas Slebodnik 2014-03-21 16:53:29 UTC
I prepared scratch build for case you don;t want (cannot) modify entries in LDAP server. SSSD will not crash if some entries contain more then one attribute "automountKey" (these entries will be skipped)

http://koji.fedoraproject.org/koji/taskinfo?taskID=6659466
and x86_64 packges
http://koji.fedoraproject.org/koji/taskinfo?taskID=6659470

Comment 9 Jakub Hrozek 2014-04-04 12:17:54 UTC
A fix for the sssd_be crash was pushed upstream in e18d5c703c529f99bd375da887d9aeaee6b82fcd

Comment 10 Fedora Update System 2014-04-08 12:23:39 UTC
sssd-1.11.5-1.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/sssd-1.11.5-1.fc19

Comment 11 Fedora Update System 2014-04-08 12:24:02 UTC
sssd-1.11.5-1.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/sssd-1.11.5-1.fc20

Comment 12 Fedora Update System 2014-04-09 13:16:01 UTC
Package sssd-1.11.5-1.fc19:
* should fix your issue,
* was pushed to the Fedora 19 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing sssd-1.11.5-1.fc19'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2014-4932/sssd-1.11.5-1.fc19
then log in and leave karma (feedback).

Comment 13 Fedora Update System 2014-04-11 18:21:32 UTC
sssd-1.11.5.1-1.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/sssd-1.11.5.1-1.fc20

Comment 14 Fedora Update System 2014-04-11 18:21:55 UTC
sssd-1.11.5.1-1.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/sssd-1.11.5.1-1.fc19

Comment 15 Fedora Update System 2014-04-24 07:36:19 UTC
sssd-1.11.5.1-1.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 16 Fedora Update System 2014-04-24 07:42:15 UTC
sssd-1.11.5.1-1.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.