Description of problem: nscd hangs on futex call and the nscd processes are using up 100% CPU on several of our machines. nscd isn't responding at all. 'service restart nscd' is not able to stop the process and nscd will only respond to a 'kill -9'. We are currently restarting nscd in a daily cronjob as a workaround. We have also noticed that on the machines where the nscd processes are using up 100% CPU, 'lsof' shows two fd:s opens /var/run/nscd/socket. But on the machines with a normal nscd 'lsof' shows only one opened /var/run/nscd/socket. This problem occurs on machines both with and without LDAP connection. Version-Release number of selected component (if applicable): nscd-2.3.4-2.41 we has also tried the patch from https://bugzilla.redhat.com/show_bug.cgi?id=428837 this didn't solve the bug. How reproducible: It occurs sporadically. But more frequently in our clusters.
I get the same problem - seems to be just 4.7 that has the problem, 4.6 and earlier are OK Previous reports about nscd using 100% CPU seem to be related to LDAP - but we don't use LDAP strace'ing the process doesn't give any useful: # strace -f -p 7448 Process 7456 attached with 7 threads - interrupt to quit [pid 7448] futex(0x552abc5500, FUTEX_WAIT, 2, NULL <unfinished ...> [pid 7451] futex(0x552abc5500, FUTEX_WAIT, 2, NULL <unfinished ...> [pid 7456] futex(0x552abc55a4, FUTEX_WAIT, 2144912, NULL <unfinished ...> [pid 7453] futex(0x552abc55a4, FUTEX_WAIT, 2144912, NULL <unfinished ...> [pid 7455] futex(0x552abc55a4, FUTEX_WAIT, 2144912, NULL <unfinished ...> [pid 7454] futex(0x552abc55a4, FUTEX_WAIT, 2144912, NULL ^C Process 7448 detached Process 7451 detached Process 7452 detached Process 7453 detached Process 7454 detached Process 7455 detached Process 7456 detached Attaching gdb to the running nscd process always gives: ... Loaded symbols for /lib64/libnss_nis.so.2 0x0000002a95790f8b in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0 (gdb) where #0 0x0000002a95790f8b in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0 #1 0x000000000000000a in ?? () #2 0x0000007fbfffd390 in ?? () #3 0x0000002a9578dd54 in pthread_mutex_lock () from /lib64/tls/libpthread.so.0 #4 0x0000000040c05960 in ?? () #5 0x0000000000000000 in ?? () (it's the same every time I run 'gdb -p pid' on the same machine) Below is the output of 'netstat -a -p | grep nscd' on one of the affected machines, along with lsof output. # netstat -a -p | grep nscd unix 2 [ ACC ] STREAM LISTENING 44664875 7448/nscd /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 2 [ ] STREAM CONNECTED 58404955 7448/nscd /var/run/nscd/socket lsof output: ntpd 4374 38 mem REG 8,1 244024 268664951 /var/db/nscd/passwd ntpd 4374 38 mem REG 8,1 217016 268664953 /var/db/nscd/group nscd 7448 28 cwd DIR 8,1 4096 64 / nscd 7448 28 rtd DIR 8,1 4096 64 / nscd 7448 28 txt REG 8,1 113944 201361698 /usr/sbin/nscd nscd 7448 28 mem REG 8,1 115168 234951758 /lib64/ld-2.3.4.so nscd 7448 28 mem REG 8,1 61336 1096 /lib64/tls/librt-2.3.4.so nscd 7448 28 mem REG 8,1 130112 1094 /lib64/tls/libpthread-2.3.4.so nscd 7448 28 mem REG 8,1 115016 234953598 /lib64/libnsl-2.3.4.so nscd 7448 28 mem REG 8,1 62536 234953604 /lib64/libselinux.so.1 nscd 7448 28 mem REG 8,1 66040 234933012 /lib64/libaudit.so.0.0.0 nscd 7448 28 mem REG 8,1 17224 234953600 /lib64/libcap.so.1.10 nscd 7448 28 mem REG 8,1 1636504 1093 /lib64/tls/libc-2.3.4.so nscd 7448 28 mem REG 8,1 48504432 33921630 /usr/lib/locale/locale-archive nscd 7448 28 mem REG 8,1 61688 234932905 /lib64/libnss_files-2.3.4.so nscd 7448 28 mem REG 8,1 63784 234881568 /lib64/libnss_nisplus-2.3.4.so nscd 7448 28 mem REG 8,1 53112 234933016 /lib64/libnss_nis-2.3.4.so nscd 7448 28 mem REG 8,1 244024 268664951 /var/db/nscd/passwd nscd 7448 28 mem REG 8,1 217016 268664953 /var/db/nscd/group nscd 7448 28 0u CHR 1,3 2369 /dev/null nscd 7448 28 1u CHR 1,3 2369 /dev/null nscd 7448 28 2u CHR 1,3 2369 /dev/null nscd 7448 28 3w REG 8,1 0 100807338 /var/log/nscd.log nscd 7448 28 4u REG 8,1 244024 268664951 /var/db/nscd/passwd nscd 7448 28 5r REG 8,1 244024 268664951 /var/db/nscd/passwd nscd 7448 28 6u REG 8,1 217016 268664953 /var/db/nscd/group nscd 7448 28 7r REG 8,1 217016 268664953 /var/db/nscd/group nscd 7448 28 8u unix 0x0000010156b32080 44664875 /var/run/nscd/socket nscd 7448 28 9r 0000 0,8 0 44664876 eventpoll nscd 7448 28 10u unix 0x0000010238d97040 58404955 /var/run/nscd/socket
Had some more machines with nscd running at 100% CPU Running 'gdb -p pid_of_nscd' gives: (gdb) info stack #0 0x0000002a95790f8b in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0 #1 0x000000000000000a in ?? () #2 0x0000007fbffff6c0 in ?? () #3 0x0000002a9578dd54 in pthread_mutex_lock () from /lib64/tls/libpthread.so.0 #4 0x0000002a9578c6ab in pthread_create@@GLIBC_2.2.5 () from /lib64/tls/libpthread.so.0 #5 0x000000552aab1983 in main () from /usr/sbin/nscd (gdb) info threads 11 Thread 1075841376 (LWP 4350) 0x0000002a95790f8b in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0 10 Thread 1077942624 (LWP 4351) 0x000000552aab0e0d in main () from /usr/sbin/nscd 9 Thread 1080043872 (LWP 4352) 0x0000002a9578eb3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/tls/libpthread.so.0 8 Thread 1082145120 (LWP 4353) 0x0000002a9578eb3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/tls/libpthread.so.0 7 Thread 1084246368 (LWP 4354) 0x0000002a9578eb3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/tls/libpthread.so.0 6 Thread 1086347616 (LWP 4355) 0x0000002a9578eb3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/tls/libpthread.so.0 5 Thread 1088448864 (LWP 5277) 0x0000002a9578eb3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/tls/libpthread.so.0 4 Thread 1090550112 (LWP 5278) 0x0000002a9578eb3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/tls/libpthread.so.0 3 Thread 1092651360 (LWP 5282) 0x0000002a9578eb3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/tls/libpthread.so.0 2 Thread 1094752608 (LWP 5283) 0x0000002a9578eb3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/tls/libpthread.so.0 1 Thread 182904237312 (LWP 4331) 0x0000002a95790f8b in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0 Looking at the glibc RPM changelog, there was a patch added since 4.6 (glibc-2.3.4-2.39) which is: fix x86_64 pthread_mutex_timedlock (#434601) Could this patch be part of the problem?
*** This bug has been marked as a duplicate of bug 495082 ***