Description of problem: Seeing a few possibly interrelated issues; 1) Occasionally, after restarting my PC (Lenovo X1 Carbon) uid/password validation may take minutes to complete. 2) dmesg displaying the following message; [ 379.671088] sssd_be[5292]: segfault at 8 ip 00007f57d687fd14 sp 00007ffc3c85c0c8 error 4 in libdbus-1.so.3.19.11[7f57d6859000+31000] 3) files in /var/log/sssd eventually filling up my filesystem Notably example is the following file; [root@vkreuk sssd]# ls -al sssd_implicit_files.log -rw-------. 1 root root 24312814498 Jun 11 09:33 sssd_implicit_files.log Looking at the file it is jam full with messages like the following (Mon May 11 16:10:38 2020) [sssd[be[implicit_files]]] [ldb] (0x0010): Failure during prepare_write): IO Error -> Protocol error (Mon May 11 16:10:38 2020) [sssd[be[implicit_files]]] [ldb] (0x0010): ltdb: tdb(/var/lib/sss/db/timestamps_implicit_files.ldb): transaction_write: failed at off=4294963200 len=2698 (Mon May 11 16:10:38 2020) [sssd[be[implicit_files]]] [ldb] (0x0010): ltdb: tdb(/var/lib/sss/db/timestamps_implicit_files.ldb): transaction_write: failed at off=4294967291 len=4 (Mon May 11 16:10:38 2020) [sssd[be[implicit_files]]] [ldb] (0x0010): ltdb: tdb(/var/lib/sss/db/timestamps_implicit_files.ldb): tdb_free: update_tailer failed! Version-Release number of selected component (if applicable): sssd-common-2.2.3-13.fc31.x86_64 How reproducible: The segfault messages seem to appear after every reboot. The minute long login is intermittent. /var/log/sssd grows steady until I get a warning that the filesystem is running out of space. Deleting the log files fixes 'stabilizes' my system Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Hi, crash looks similart to bz 1783169 but difficult to say with information provided. could you please: 1) attach coredump 2) check if sudo: ``` systemctl stop sssd rm -rf /var/lib/sss/db/* systemctl start sssd ``` helps?
Created attachment 1697957 [details] Most recent sssd core dump Most recent core dump
Created attachment 1697959 [details] file containing output of 'coredumpctl list' Output of 'coredumpctl list'. Many, many core dumps due to sssd
Hi Alexey - I attached a file containing the output of 'coredumpctl list' and the most recent core dump to this case. Next step is to issue the following commands and see if it resolves the issue. systemctl stop sssd rm -rf /var/lib/sss/db/* systemctl start sssd Will keep you posted. Thanks for your help. regards, Volkert
Apparently file /etc/krb5.keytab is missing (see below). Might be the culprit. Will create the file and restart again. [vkreuk@vkreuk campus]$ sudo systemctl start sssd [sudo] password for vkreuk: Job for sssd.service failed because the control process exited with error code. See "systemctl status sssd.service" and "journalctl -xe" for details. [vkreuk@vkreuk campus]$ sudo systemctl status sssd.service ● sssd.service - System Security Services Daemon Loaded: loaded (/usr/lib/systemd/system/sssd.service; enabled; vendor preset: enabled) Active: activating (start) since Thu 2020-06-18 10:45:29 EDT; 732ms ago Main PID: 426490 (sssd) Tasks: 2 (limit: 19007) Memory: 6.4M CPU: 215ms CGroup: /system.slice/sssd.service ├─426490 /usr/sbin/sssd -i --logger=files └─426491 /usr/libexec/sssd/sssd_be --domain implicit_files --uid 0 --gid 0 --logger=files Jun 18 10:45:29 vkreuk.users.ipa.redhat.com sssd[be[ipa.redhat.com]][426492]: Failed to read keytab [FILE:/etc/krb5.keytab]: No suitable principal found in k> Jun 18 10:45:29 vkreuk.users.ipa.redhat.com sssd[be[ipa.redhat.com]][426493]: Starting up Jun 18 10:45:29 vkreuk.users.ipa.redhat.com sssd[be[ipa.redhat.com]][426493]: krb5_kt_start_seq_get failed: Key table file '/etc/krb5.keytab' not found Jun 18 10:45:29 vkreuk.users.ipa.redhat.com sssd[be[ipa.redhat.com]][426493]: krb5_kt_start_seq_get failed: Key table file '/etc/krb5.keytab' not found Jun 18 10:45:29 vkreuk.users.ipa.redhat.com sssd[be[ipa.redhat.com]][426493]: krb5_kt_start_seq_get failed: Key table file '/etc/krb5.keytab' not found Jun 18 10:45:29 vkreuk.users.ipa.redhat.com sssd[be[ipa.redhat.com]][426493]: krb5_kt_start_seq_get failed: Key table file '/etc/krb5.keytab' not found Jun 18 10:45:29 vkreuk.users.ipa.redhat.com sssd[be[ipa.redhat.com]][426493]: krb5_kt_start_seq_get failed: Key table file '/etc/krb5.keytab' not found Jun 18 10:45:29 vkreuk.users.ipa.redhat.com sssd[be[ipa.redhat.com]][426493]: krb5_kt_start_seq_get failed: Key table file '/etc/krb5.keytab' not found Jun 18 10:45:29 vkreuk.users.ipa.redhat.com sssd[be[ipa.redhat.com]][426493]: krb5_kt_start_seq_get failed: Key table file '/etc/krb5.keytab' not found Jun 18 10:45:29 vkreuk.users.ipa.redhat.com sssd[be[ipa.redhat.com]][426493]: Failed to read keytab [FILE:/etc/krb5.keytab]: No suitable principal found in k> [vkreuk@vkreuk campus]$ ls -al /etc/krb5.keytab ls: cannot access '/etc/krb5.keytab': No such file or directory
Looks like you are using IPA provider by machine is not enrolled to the domain?
> Most recent sssd core dump (314 bytes, application/x-desktop) This doesn't look like a coredump... Perhaps you attached a wrong file?
Hi Alexey - are you looking at core.sssd_be.0.6bb0860ae6444238bcbb3ae058ee3eaa.382249.1592482040000000000000.lz4? Belief it's > 900KB. Note that the 'other' file is simple text file listing all core dumps. Note - I am working with our security people to get a proper krb5.keytab file. You may want to defer the analysis of the core dump until I let you know whether this makes any changes. Thanks. Volkert
Created attachment 1697985 [details] updated core dump file
ps. 'fixed' the core dump file.
Missing keytab shouldn't be a reason of a crash. It seems I don't have the same system at hand to obtain backtrace. Could you please run on your system: ``` gdb /usr/libexec/sssd/sssd_be coredump bt ``` and provide output?
[vkreuk@vkreuk campus]$ su Password: [root@vkreuk campus]# gdb /usr/libexec/sssd/sssd_be coredump GNU gdb (GDB) Fedora 8.3.50.20190824-30.fc31 Copyright (C) 2019 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/libexec/sssd/sssd_be... Reading symbols from .gnu_debugdata for /usr/libexec/sssd/sssd_be... (No debugging symbols found in .gnu_debugdata for /usr/libexec/sssd/sssd_be) /home/vkreuk/acsgit/NetConfig/campus/coredump: No such file or directory. Missing separate debuginfos, use: dnf debuginfo-install sssd-common-2.2.3-13.fc31.x86_64 (gdb) bt No stack. (gdb)
here it is; [root@vkreuk campus]# gdb /usr/libexec/sssd/sssd_be coredump GNU gdb (GDB) Fedora 8.3.50.20190824-30.fc31 Copyright (C) 2019 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/libexec/sssd/sssd_be... Reading symbols from .gnu_debugdata for /usr/libexec/sssd/sssd_be... (No debugging symbols found in .gnu_debugdata for /usr/libexec/sssd/sssd_be) /home/vkreuk/acsgit/NetConfig/campus/coredump: No such file or directory. Missing separate debuginfos, use: dnf debuginfo-install sssd-common-2.2.3-13.fc31.x86_64 (gdb) bt No stack. (gdb) exit Undefined command: "exit". Try "help". (gdb) quit
with the help of a colleague I reinstalled my ipa client. After rebooting my workstation is do not see core dump messages during restart. Only (sssd) message I see is as follows. Not sure if it's an issue. [vkreuk@vkreuk ~]$ dmesg | grep sssd [ 16.490312] systemd[1]: /usr/lib/systemd/system/sssd.service:12: PIDFile= references a path below legacy directory /var/run/, updating /var/run/sssd.pid → /run/sssd.pid; please update the unit file accordingly. In addition - 'coredumpctl list' suggests my system has stopped its continuous core dumping and the sssd log files in /var/log/sssd/ have stopped their rebellious growth; i.e. tail -f sssd_implicit_files.log and tail -f sssd.log no longer show continuous messages. Not 100% sure if the occasional time-consuming login has disappeared as well. Hasn't happened so far. Will let you know if it shows its ugly head again. Ergo - it seems I am ok now. Thanks for your help.
(In reply to vkreuk from comment #13) > here it is; > > [root@vkreuk campus]# gdb /usr/libexec/sssd/sssd_be coredump ... > /home/vkreuk/acsgit/NetConfig/campus/coredump: No such file or director > Missing separate debuginfos, use: dnf debuginfo-install sssd-common-2.2.3-13.fc31.x86_64 > (gdb) bt > No stack. Under "coredump" I meant path to the coredump file that you attached to the BZ. (In reply to vkreuk from comment #14) > with the help of a colleague I reinstalled my ipa client. After rebooting > my workstation is do not see core dump messages during restart. ... > Ergo - it seems I am ok now. Thanks for your help. Glad to hear your issue was resolved. Still it would be good to figure out the reason of the crash. Besides wrong path to the coredump file, I see debug info package isn't installed on your system. Could you please: `sudo dnf debuginfo-install sssd` and then ``` gdb /usr/libexec/sssd/sssd_be "path_to_coredump_that_was_attached" bt ``` and share output? Also, could you please verify `sssd_implicit_files.log` doesn't grow due to excessive [ldb] logs?
[root@vkreuk vkreuk]# gdb /usr/libexec/sssd/sssd_be /home/vkreuk/core.sssd_be.0.6bb0860ae6444238bcbb3ae058ee3eaa.382249.1592482040000000000000 GNU gdb (GDB) Fedora 8.3.50.20190824-30.fc31 Copyright (C) 2019 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/libexec/sssd/sssd_be... Missing separate debuginfo for /usr/libexec/sssd/sssd_be Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/3b/db6eafee7056f536251ff8b600913d022837c6.debug Reading symbols from .gnu_debugdata for /usr/libexec/sssd/sssd_be... (No debugging symbols found in .gnu_debugdata for /usr/libexec/sssd/sssd_be) [New LWP 382249] Missing separate debuginfo for /usr/lib64/sssd/libsss_util.so Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/58/4e293995b8edf58d7ffea995e756fe659d4b95.debug Missing separate debuginfo for /usr/lib64/sssd/libsss_child.so Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/34/27d625fc6a3d35e97edc8028043e62b1e6cd5a.debug Missing separate debuginfo for /usr/lib64/sssd/libsss_debug.so Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/57/28bf3468ece7dfd040dd30e73b927140894561.debug Missing separate debuginfo for /usr/lib64/sssd/libsss_iface.so Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/f9/01923dd72c8c93d287c8cacc34baab10673909.debug Missing separate debuginfo for /usr/lib64/sssd/libsss_sbus.so Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/39/2a90a419f7cd0586b89eb886d82b45bba63d1b.debug Missing separate debuginfo for /usr/lib64/sssd/libsss_cert.so Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/59/fa84cbf19f46a9b2c5aaa585aa3cef4b125c70.debug Missing separate debuginfo for /usr/lib64/sssd/libsss_crypt.so Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/45/d5b2faf3b280efca33e7574ec7df67e813f0ff.debug Missing separate debuginfo for /lib64/libsss_certmap.so.0 Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/55/ccf4b5647194f18f23765e0b779ce6eb40c056.debug [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Missing separate debuginfo for /usr/lib64/ldb/modules/ldb/memberof.so Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/e8/6174c9e06eb5a6b59853333f6794d89fd1a363.debug Missing separate debuginfo for /lib64/libnss_sss.so.2 Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/93/57856aaef3f2a18bbbb3788c5a158de7beef35.debug Missing separate debuginfo for /usr/lib64/sssd/libsss_files.so Try: dnf --enablerepo='*debug*' install /usr/lib/debug/.build-id/3b/fb46fdc40bae3d86f9ff9632a7493f261ff453.debug Core was generated by `/usr/libexec/sssd/sssd_be --domain implicit_files --uid 0 --gid 0 --logger=file'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f4452cb4d14 in _dbus_list_unlink () from /lib64/libdbus-1.so.3 Missing separate debuginfos, use: dnf debuginfo-install audit-libs-3.0-0.15.20191104git1c2f876.fc31.x86_64 c-ares-1.15.0-4.fc31.x86_64 cyrus-sasl-lib-2.1.27-3.fc31.x86_64 dbus-libs-1.12.16-3.fc31.x86_64 glib2-2.62.6-1.fc31.x86_64 glibc-2.30-11.fc31.x86_64 keyutils-libs-1.6-3.fc31.x86_64 krb5-libs-1.17-46.fc31.x86_64 libbasicobjects-0.1.1-43.fc31.x86_64 libcap-ng-0.7.10-1.fc31.x86_64 libcollection-0.7.0-43.fc31.x86_64 libcom_err-1.45.5-1.fc31.x86_64 libdhash-0.5.0-43.fc31.x86_64 libgcc-9.3.1-2.fc31.x86_64 libgcrypt-1.8.5-1.fc31.x86_64 libini_config-1.3.1-43.fc31.x86_64 libldb-2.0.10-1.fc31.x86_64 libpath_utils-0.2.1-43.fc31.x86_64 libref_array-0.1.5-43.fc31.x86_64 libselinux-2.9-5.fc31.x86_64 libtalloc-2.3.0-1.fc31.x86_64 libtdb-1.4.2-1.fc31.x86_64 libtevent-0.10.1-1.fc31.x86_64 libxcrypt-4.4.16-3.fc31.x86_64 lmdb-libs-0.9.24-1.fc31.x86_64 lz4-libs-1.9.1-1.fc31.x86_64 openldap-2.4.47-3.fc31.x86_64 pam-1.3.1-21.fc31.x86_64 pcre-8.44-1.fc31.x86_64 pcre2-10.35-1.fc31.x86_64 popt-1.16-18.fc31.x86_64 zlib-1.2.11-20.fc31.x86_64 (gdb) bt #0 0x00007f4452cb4d14 in _dbus_list_unlink () from /lib64/libdbus-1.so.3 #1 0x00007f4452cb4d8d in _dbus_list_remove_link () from /lib64/libdbus-1.so.3 #2 0x00007f4452ca35f5 in _dbus_message_remove_counter () from /lib64/libdbus-1.so.3 #3 0x00007f4452c97004 in _dbus_connection_get_dispatch_status_unlocked () from /lib64/libdbus-1.so.3 #4 0x00007f4452c98cd2 in _dbus_connection_handle_watch () from /lib64/libdbus-1.so.3 #5 0x00007f4452eec49c in sbus_watch_handler () from /usr/lib64/sssd/libsss_sbus.so #6 0x00007f4452ec6f11 in tevent_common_invoke_fd_handler () from /lib64/libtevent.so.0 #7 0x00007f4452ecd3d7 in epoll_event_loop_once () from /lib64/libtevent.so.0 #8 0x00007f4452ecb53b in std_event_loop_once () from /lib64/libtevent.so.0 #9 0x00007f4452ec65d8 in _tevent_loop_once () from /lib64/libtevent.so.0 #10 0x00007f4452ec687b in tevent_common_loop_wait () from /lib64/libtevent.so.0 #11 0x00007f4452ecb4cb in std_event_loop_wait () from /lib64/libtevent.so.0 #12 0x00007f4452fe2397 in server_loop () from /usr/lib64/sssd/libsss_util.so #13 0x000055d8553dfc22 in main ()
Also - /var/log/sssd/sssd_implicit_files.log is no longer growing.
(In reply to vkreuk from comment #16) > (gdb) bt > #0 0x00007f4452cb4d14 in _dbus_list_unlink () from /lib64/libdbus-1.so.3 > #1 0x00007f4452cb4d8d in _dbus_list_remove_link () from /lib64/libdbus-1.so.3 > #2 0x00007f4452ca35f5 in _dbus_message_remove_counter () from /lib64/libdbus-1.so.3 Thank you. I believe this is a duplicate of bz 1783169 (In reply to vkreuk from comment #17) > Also - /var/log/sssd/sssd_implicit_files.log is no longer growing. Thanks for the confirmation. *** This bug has been marked as a duplicate of bug 1783169 ***