Bug 492581
Summary: | nscd uses 100% of CPU | ||
---|---|---|---|
Product: | Red Hat Enterprise Linux 4 | Reporter: | Malin <malin.frohm> |
Component: | glibc | Assignee: | Andreas Schwab <schwab> |
Status: | CLOSED DUPLICATE | QA Contact: | BaseOS QE <qe-baseos-auto> |
Severity: | high | Docs Contact: | |
Priority: | low | ||
Version: | 4.7 | CC: | cevich, djuran, jakub, james-p, ofourdan, paul.seymour, sputhenp |
Target Milestone: | --- | ||
Target Release: | --- | ||
Hardware: | All | ||
OS: | Linux | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | Bug Fix | |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2009-09-07 12:46:29 UTC | Type: | --- |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Malin
2009-03-27 14:52:04 UTC
I get the same problem - seems to be just 4.7 that has the problem, 4.6 and earlier are OK Previous reports about nscd using 100% CPU seem to be related to LDAP - but we don't use LDAP strace'ing the process doesn't give any useful: # strace -f -p 7448 Process 7456 attached with 7 threads - interrupt to quit [pid 7448] futex(0x552abc5500, FUTEX_WAIT, 2, NULL <unfinished ...> [pid 7451] futex(0x552abc5500, FUTEX_WAIT, 2, NULL <unfinished ...> [pid 7456] futex(0x552abc55a4, FUTEX_WAIT, 2144912, NULL <unfinished ...> [pid 7453] futex(0x552abc55a4, FUTEX_WAIT, 2144912, NULL <unfinished ...> [pid 7455] futex(0x552abc55a4, FUTEX_WAIT, 2144912, NULL <unfinished ...> [pid 7454] futex(0x552abc55a4, FUTEX_WAIT, 2144912, NULL ^C Process 7448 detached Process 7451 detached Process 7452 detached Process 7453 detached Process 7454 detached Process 7455 detached Process 7456 detached Attaching gdb to the running nscd process always gives: ... Loaded symbols for /lib64/libnss_nis.so.2 0x0000002a95790f8b in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0 (gdb) where #0 0x0000002a95790f8b in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0 #1 0x000000000000000a in ?? () #2 0x0000007fbfffd390 in ?? () #3 0x0000002a9578dd54 in pthread_mutex_lock () from /lib64/tls/libpthread.so.0 #4 0x0000000040c05960 in ?? () #5 0x0000000000000000 in ?? () (it's the same every time I run 'gdb -p pid' on the same machine) Below is the output of 'netstat -a -p | grep nscd' on one of the affected machines, along with lsof output. # netstat -a -p | grep nscd unix 2 [ ACC ] STREAM LISTENING 44664875 7448/nscd /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 3 [ ] STREAM CONNECTING 0 - /var/run/nscd/socket unix 2 [ ] STREAM CONNECTED 58404955 7448/nscd /var/run/nscd/socket lsof output: ntpd 4374 38 mem REG 8,1 244024 268664951 /var/db/nscd/passwd ntpd 4374 38 mem REG 8,1 217016 268664953 /var/db/nscd/group nscd 7448 28 cwd DIR 8,1 4096 64 / nscd 7448 28 rtd DIR 8,1 4096 64 / nscd 7448 28 txt REG 8,1 113944 201361698 /usr/sbin/nscd nscd 7448 28 mem REG 8,1 115168 234951758 /lib64/ld-2.3.4.so nscd 7448 28 mem REG 8,1 61336 1096 /lib64/tls/librt-2.3.4.so nscd 7448 28 mem REG 8,1 130112 1094 /lib64/tls/libpthread-2.3.4.so nscd 7448 28 mem REG 8,1 115016 234953598 /lib64/libnsl-2.3.4.so nscd 7448 28 mem REG 8,1 62536 234953604 /lib64/libselinux.so.1 nscd 7448 28 mem REG 8,1 66040 234933012 /lib64/libaudit.so.0.0.0 nscd 7448 28 mem REG 8,1 17224 234953600 /lib64/libcap.so.1.10 nscd 7448 28 mem REG 8,1 1636504 1093 /lib64/tls/libc-2.3.4.so nscd 7448 28 mem REG 8,1 48504432 33921630 /usr/lib/locale/locale-archive nscd 7448 28 mem REG 8,1 61688 234932905 /lib64/libnss_files-2.3.4.so nscd 7448 28 mem REG 8,1 63784 234881568 /lib64/libnss_nisplus-2.3.4.so nscd 7448 28 mem REG 8,1 53112 234933016 /lib64/libnss_nis-2.3.4.so nscd 7448 28 mem REG 8,1 244024 268664951 /var/db/nscd/passwd nscd 7448 28 mem REG 8,1 217016 268664953 /var/db/nscd/group nscd 7448 28 0u CHR 1,3 2369 /dev/null nscd 7448 28 1u CHR 1,3 2369 /dev/null nscd 7448 28 2u CHR 1,3 2369 /dev/null nscd 7448 28 3w REG 8,1 0 100807338 /var/log/nscd.log nscd 7448 28 4u REG 8,1 244024 268664951 /var/db/nscd/passwd nscd 7448 28 5r REG 8,1 244024 268664951 /var/db/nscd/passwd nscd 7448 28 6u REG 8,1 217016 268664953 /var/db/nscd/group nscd 7448 28 7r REG 8,1 217016 268664953 /var/db/nscd/group nscd 7448 28 8u unix 0x0000010156b32080 44664875 /var/run/nscd/socket nscd 7448 28 9r 0000 0,8 0 44664876 eventpoll nscd 7448 28 10u unix 0x0000010238d97040 58404955 /var/run/nscd/socket Had some more machines with nscd running at 100% CPU Running 'gdb -p pid_of_nscd' gives: (gdb) info stack #0 0x0000002a95790f8b in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0 #1 0x000000000000000a in ?? () #2 0x0000007fbffff6c0 in ?? () #3 0x0000002a9578dd54 in pthread_mutex_lock () from /lib64/tls/libpthread.so.0 #4 0x0000002a9578c6ab in pthread_create@@GLIBC_2.2.5 () from /lib64/tls/libpthread.so.0 #5 0x000000552aab1983 in main () from /usr/sbin/nscd (gdb) info threads 11 Thread 1075841376 (LWP 4350) 0x0000002a95790f8b in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0 10 Thread 1077942624 (LWP 4351) 0x000000552aab0e0d in main () from /usr/sbin/nscd 9 Thread 1080043872 (LWP 4352) 0x0000002a9578eb3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/tls/libpthread.so.0 8 Thread 1082145120 (LWP 4353) 0x0000002a9578eb3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/tls/libpthread.so.0 7 Thread 1084246368 (LWP 4354) 0x0000002a9578eb3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/tls/libpthread.so.0 6 Thread 1086347616 (LWP 4355) 0x0000002a9578eb3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/tls/libpthread.so.0 5 Thread 1088448864 (LWP 5277) 0x0000002a9578eb3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/tls/libpthread.so.0 4 Thread 1090550112 (LWP 5278) 0x0000002a9578eb3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/tls/libpthread.so.0 3 Thread 1092651360 (LWP 5282) 0x0000002a9578eb3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/tls/libpthread.so.0 2 Thread 1094752608 (LWP 5283) 0x0000002a9578eb3a in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/tls/libpthread.so.0 1 Thread 182904237312 (LWP 4331) 0x0000002a95790f8b in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0 Looking at the glibc RPM changelog, there was a patch added since 4.6 (glibc-2.3.4-2.39) which is: fix x86_64 pthread_mutex_timedlock (#434601) Could this patch be part of the problem? *** This bug has been marked as a duplicate of bug 495082 *** |