Bug 492581 - nscd uses 100% of CPU
nscd uses 100% of CPU
Status: CLOSED DUPLICATE of bug 495082
Product: Red Hat Enterprise Linux 4
Classification: Red Hat
Component: glibc (Show other bugs)
4.7
All Linux
low Severity high
: ---
: ---
Assigned To: Andreas Schwab
BaseOS QE
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2009-03-27 10:52 EDT by Malin
Modified: 2009-09-07 08:46 EDT (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2009-09-07 08:46:29 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
CentOS 3373 None None None Never

  None (edit)
Description Malin 2009-03-27 10:52:04 EDT
Description of problem:
nscd hangs on futex call and the nscd processes are using up 100% CPU on several of our machines.  nscd isn't responding at all. 'service restart nscd' is not able to stop the process and nscd will only respond to a 'kill -9'. We are currently restarting nscd in a daily cronjob as a workaround.
 
We have also noticed that on the machines where the nscd processes are using up 100% CPU, 'lsof' shows two fd:s opens /var/run/nscd/socket. But on the machines with a normal nscd 'lsof' shows only one opened /var/run/nscd/socket.

This problem occurs on machines both with and without LDAP connection.

Version-Release number of selected component (if applicable):
nscd-2.3.4-2.41
we has also tried the patch from https://bugzilla.redhat.com/show_bug.cgi?id=428837 this didn't solve the bug.

How reproducible:
It occurs sporadically. But more frequently in our clusters.
Comment 1 James Pearson 2009-03-28 04:46:14 EDT
I get the same problem - seems to be just 4.7 that has the problem, 4.6 and earlier are OK

Previous reports about nscd using 100% CPU seem to be related to LDAP - but we don't use LDAP

strace'ing the process doesn't give any useful:

# strace -f -p 7448
Process 7456 attached with 7 threads - interrupt to quit
[pid  7448] futex(0x552abc5500, FUTEX_WAIT, 2, NULL <unfinished ...>
[pid  7451] futex(0x552abc5500, FUTEX_WAIT, 2, NULL <unfinished ...>
[pid  7456] futex(0x552abc55a4, FUTEX_WAIT, 2144912, NULL <unfinished ...>
[pid  7453] futex(0x552abc55a4, FUTEX_WAIT, 2144912, NULL <unfinished ...>
[pid  7455] futex(0x552abc55a4, FUTEX_WAIT, 2144912, NULL <unfinished ...>
[pid  7454] futex(0x552abc55a4, FUTEX_WAIT, 2144912, NULL
^C
Process 7448 detached
Process 7451 detached
Process 7452 detached
Process 7453 detached
Process 7454 detached
Process 7455 detached
Process 7456 detached

Attaching gdb to the running nscd process always gives:

...
Loaded symbols for /lib64/libnss_nis.so.2
0x0000002a95790f8b in __lll_mutex_lock_wait ()
   from /lib64/tls/libpthread.so.0
(gdb) where
#0  0x0000002a95790f8b in __lll_mutex_lock_wait ()
   from /lib64/tls/libpthread.so.0
#1  0x000000000000000a in ?? ()
#2  0x0000007fbfffd390 in ?? ()
#3  0x0000002a9578dd54 in pthread_mutex_lock () from /lib64/tls/libpthread.so.0
#4  0x0000000040c05960 in ?? ()
#5  0x0000000000000000 in ?? ()

(it's the same every time I run 'gdb -p pid' on the same machine)

Below is the output of 'netstat -a -p | grep nscd' on one of the affected machines, along with lsof output.

# netstat -a -p | grep nscd
unix  2      [ ACC ]     STREAM     LISTENING     44664875 7448/nscd           /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  3      [ ]         STREAM     CONNECTING    0      -                   /var/run/nscd/socket
unix  2      [ ]         STREAM     CONNECTED     58404955 7448/nscd           /var/run/nscd/socket

lsof output:

ntpd    4374       38  mem       REG                8,1   244024 268664951 /var/db/nscd/passwd
ntpd    4374       38  mem       REG                8,1   217016 268664953 /var/db/nscd/group
nscd    7448       28  cwd       DIR                8,1     4096        64 /
nscd    7448       28  rtd       DIR                8,1     4096        64 /
nscd    7448       28  txt       REG                8,1   113944 201361698 /usr/sbin/nscd
nscd    7448       28  mem       REG                8,1   115168 234951758 /lib64/ld-2.3.4.so
nscd    7448       28  mem       REG                8,1    61336      1096 /lib64/tls/librt-2.3.4.so
nscd    7448       28  mem       REG                8,1   130112      1094 /lib64/tls/libpthread-2.3.4.so
nscd    7448       28  mem       REG                8,1   115016 234953598 /lib64/libnsl-2.3.4.so
nscd    7448       28  mem       REG                8,1    62536 234953604 /lib64/libselinux.so.1
nscd    7448       28  mem       REG                8,1    66040 234933012 /lib64/libaudit.so.0.0.0
nscd    7448       28  mem       REG                8,1    17224 234953600 /lib64/libcap.so.1.10
nscd    7448       28  mem       REG                8,1  1636504      1093 /lib64/tls/libc-2.3.4.so
nscd    7448       28  mem       REG                8,1 48504432  33921630 /usr/lib/locale/locale-archive
nscd    7448       28  mem       REG                8,1    61688 234932905 /lib64/libnss_files-2.3.4.so
nscd    7448       28  mem       REG                8,1    63784 234881568 /lib64/libnss_nisplus-2.3.4.so
nscd    7448       28  mem       REG                8,1    53112 234933016 /lib64/libnss_nis-2.3.4.so
nscd    7448       28  mem       REG                8,1   244024 268664951 /var/db/nscd/passwd
nscd    7448       28  mem       REG                8,1   217016 268664953 /var/db/nscd/group
nscd    7448       28    0u      CHR                1,3               2369 /dev/null
nscd    7448       28    1u      CHR                1,3               2369 /dev/null
nscd    7448       28    2u      CHR                1,3               2369 /dev/null
nscd    7448       28    3w      REG                8,1        0 100807338 /var/log/nscd.log
nscd    7448       28    4u      REG                8,1   244024 268664951 /var/db/nscd/passwd
nscd    7448       28    5r      REG                8,1   244024 268664951 /var/db/nscd/passwd
nscd    7448       28    6u      REG                8,1   217016 268664953 /var/db/nscd/group
nscd    7448       28    7r      REG                8,1   217016 268664953 /var/db/nscd/group
nscd    7448       28    8u     unix 0x0000010156b32080           44664875 /var/run/nscd/socket
nscd    7448       28    9r     0000                0,8        0  44664876 eventpoll
nscd    7448       28   10u     unix 0x0000010238d97040           58404955 /var/run/nscd/socket
Comment 2 James Pearson 2009-03-30 07:01:23 EDT
Had some more machines with nscd running at 100% CPU

Running 'gdb -p pid_of_nscd' gives:

(gdb) info stack
#0  0x0000002a95790f8b in __lll_mutex_lock_wait ()
   from /lib64/tls/libpthread.so.0
#1  0x000000000000000a in ?? ()
#2  0x0000007fbffff6c0 in ?? ()
#3  0x0000002a9578dd54 in pthread_mutex_lock () from /lib64/tls/libpthread.so.0
#4  0x0000002a9578c6ab in pthread_create@@GLIBC_2.2.5 ()
   from /lib64/tls/libpthread.so.0
#5  0x000000552aab1983 in main () from /usr/sbin/nscd
(gdb) info threads
  11 Thread 1075841376 (LWP 4350)  0x0000002a95790f8b in __lll_mutex_lock_wait
    () from /lib64/tls/libpthread.so.0
  10 Thread 1077942624 (LWP 4351)  0x000000552aab0e0d in main ()
   from /usr/sbin/nscd
  9 Thread 1080043872 (LWP 4352)  0x0000002a9578eb3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/tls/libpthread.so.0
  8 Thread 1082145120 (LWP 4353)  0x0000002a9578eb3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/tls/libpthread.so.0
  7 Thread 1084246368 (LWP 4354)  0x0000002a9578eb3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/tls/libpthread.so.0
  6 Thread 1086347616 (LWP 4355)  0x0000002a9578eb3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/tls/libpthread.so.0
  5 Thread 1088448864 (LWP 5277)  0x0000002a9578eb3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/tls/libpthread.so.0
  4 Thread 1090550112 (LWP 5278)  0x0000002a9578eb3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/tls/libpthread.so.0
  3 Thread 1092651360 (LWP 5282)  0x0000002a9578eb3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/tls/libpthread.so.0
  2 Thread 1094752608 (LWP 5283)  0x0000002a9578eb3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/tls/libpthread.so.0
  1 Thread 182904237312 (LWP 4331)  0x0000002a95790f8b in __lll_mutex_lock_wait
    () from /lib64/tls/libpthread.so.0

Looking at the glibc RPM changelog, there was a patch added since 4.6 (glibc-2.3.4-2.39) which is:

 fix x86_64 pthread_mutex_timedlock (#434601)

Could this patch be part of the problem?
Comment 4 Andreas Schwab 2009-09-07 08:46:29 EDT

*** This bug has been marked as a duplicate of bug 495082 ***

Note You need to log in before you can comment on or make changes to this bug.