Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Created attachment 1353272[details]
valgrind output
Description of problem:
After the winbind package upgraded from samba-winbind-3.6.23-43.el6_9 to samba-winbind-3.6.23-45.el6_9, Customer noticed it consumes huge memory and then oom-killer is invoked and kills the process.
$ dmesg|grep -i killed
Killed process pid, UID 0, (winbindd) total-vm:206912kB, anon-rss:572kB, file-rss:124kB
Killed process pid3, UID 0, (winbindd) total-vm:204672kB, anon-rss:296kB, file-rss:788kB
Killed process pid4, UID 0, (winbindd) total-vm:204836kB, anon-rss:292kB, file-rss:812kB
Killed process pid7, UID 0, (winbindd) total-vm:204836kB, anon-rss:292kB, file-rss:704kB
If customer downgrades winbind package to:
samba-winbind-3.6.23-43.el6_9.x86_64.rpm
Issue is not seen.
Issue seen with these packages:
# rpm -qa|grep samba
samba-winbind-3.6.23-45.el6_9.x86_64
samba4-libs-4.2.10-11.el6_9.x86_64
samba-common-3.6.23-45.el6_9.x86_64
samba-client-3.6.23-45.el6_9.x86_64
samba-winbind-clients-3.6.23-45.el6_9.x86_64
No Issues with these packages:
libsmbclient-3.6.23-43.el6_9.x86_64.rpm
samba4-libs-4.2.10-10.el6_9.x86_64.rpm
samba-client-3.6.23-43.el6_9.x86_64.rpm
samba-common-3.6.23-43.el6_9.x86_64.rpm
samba-winbind-3.6.23-43.el6_9.x86_64.rpm
samba-winbind-clients-3.6.23-43.el6_9.x86_64.rpm
Other Info:
1. selinux is disabled.
2. Issue persists even if no AD user logs into samba server.
3. Has winbind offline logon = false in smb.conf
4. Issue reappears after a while of restarting the winbind service no need for user to login.
5. Customer is ready to test the test-rpms.
Version-Release number of selected component (if applicable):
samba-winbind-3.6.23-45.el6_9.x86_64
How reproducible:
All times in customer env.
Steps to Reproduce:
1. Set up samba-6.5 authenticating to AD using winbind
2. winbind package should be samba-winbind-3.6.23-45.
3. start winbind and try connecting to one of the share.
4. Only connection from only 1 user also shows winbind consuming lot of memory.
Actual results:
winbind memory leak and consumes resources
Expected results:
winbind should not leak memory and consume resources
Additional info:
Hello,
Another Customer(Cargill Inc) using samba-3.6.23-45.el6_9 found winbind leak Issues.
# tail -f /var/log/samba/winbindd.log
[2017/11/16 12:30:45.012275, 0] winbindd/winbindd_dual.c:1392(fork_domain_child)
Could not fork: Cannot allocate memory
[2017/11/16 12:30:45.015812, 0] winbindd/winbindd_dual.c:1392(fork_domain_child)
Could not fork: Cannot allocate memory
[2017/11/16 12:30:45.021137, 0] winbindd/winbindd_dual.c:1392(fork_domain_child)
Could not fork: Cannot allocate memory
[2017/11/16 12:32:01.927146, 0] param/loadparm.c:8056(lp_do_parameter)
Global parameter unix extensions found in service section!
[2017/11/16 12:32:01.937565, 0] winbindd/winbindd_cache.c:3204(initialize_winbindd_cache)
initialize_winbindd_cache: clearing cache and re-creating with version number 2
# tail -f /var/log/messages
Nov 16 12:30:50 XXXX kernel: [20260] 0 20260 1026 17 2 0 0 tail
Nov 16 12:30:50 XXXX kernel: [30144] 0 30144 102022 262 3 0 0 LogMonitoring
Nov 16 12:30:50 XXXX kernel: Out of memory: Kill process 15343 (winbindd) score 637 or sacrifice child
Nov 16 12:30:50 XXXX kernel: Killed process 15343, UID 0, (winbindd) total-vm:39083488kB, anon-rss:22851136kB, file-rss:36kB
Nov 16 12:32:01 XXXX winbindd[9283]: [2017/11/16 12:32:01.927146, 0] param/loadparm.c:8056(lp_do_parameter)
Nov 16 12:32:01 XXXX winbindd[9283]: Global parameter unix extensions found in service section!
Nov 16 12:32:01 XXXX winbindd[9285]: [2017/11/16 12:32:01.937565, 0] winbindd/winbindd_cache.c:3204(initialize_winbindd_cache)
Nov 16 12:32:01 XXXX winbindd[9285]: initialize_winbindd_cache: clearing cache and re-creating with version number 2
valgrind Output is taken after running:
# /usr/bin/valgrind --trace-children=yes --show-reachable=yes --track-origins=yes --read-var-info=yes --tool=memcheck --leak-check=full --num-callers=50 -v --log-file=/tmp/valgrind.out /usr/sbin/winbindd -F -S
Thanks
Amit
Comment 4Andreas Schneider
2017-11-21 08:42:35 UTC
Please always install the 'samba-debuginfo' package before you run valgrind!
USE: valgrind --tool=memcheck -v --num-callers=20 --track-origins=yes --trace-children=yes
DO NOT ADD: --leak-check nor --show-reachables
Comment 34Andreas Schneider
2018-01-23 15:12:30 UTC
It looked like from previous information that the issue is in the main winbindd process (see 'ps afx'). Assuming that this piece of information is correct we will need the following steps:
1. Upgrade the samba packages that had the issue.
2. Install samba-debuginfo libtevent-debuginfo
3. Enable samba debug logs
service winbind stop; service smb stop; service nmb stop;
log level = 10
debug pid = true
max log size = 0
rm -f /var/log/samba/*
service smb start; service nmb start
4. valgrind --tool=memcheck -v --num-callers=20 --track-origins=yes --leak-check=full /usr/sbin/winbindd and wait for the issue to happen (the memory of winbindd starts growing and growing)
5. Provide samba and valgrind logs including the output of 'wbinfo --trusted-domains --verbose"'
Comment 36Andreas Schneider
2018-02-08 11:20:29 UTC
Where can I find the logs?
Comment 43Andreas Schneider
2018-02-20 12:48:09 UTC
Could I get the coredump please?
Comment 59Andreas Schneider
2018-03-21 16:36:10 UTC
*** Bug 1558933 has been marked as a duplicate of this bug. ***
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHSA-2018:1860
Created attachment 1353272 [details] valgrind output Description of problem: After the winbind package upgraded from samba-winbind-3.6.23-43.el6_9 to samba-winbind-3.6.23-45.el6_9, Customer noticed it consumes huge memory and then oom-killer is invoked and kills the process. $ dmesg|grep -i killed Killed process pid, UID 0, (winbindd) total-vm:206912kB, anon-rss:572kB, file-rss:124kB Killed process pid3, UID 0, (winbindd) total-vm:204672kB, anon-rss:296kB, file-rss:788kB Killed process pid4, UID 0, (winbindd) total-vm:204836kB, anon-rss:292kB, file-rss:812kB Killed process pid7, UID 0, (winbindd) total-vm:204836kB, anon-rss:292kB, file-rss:704kB If customer downgrades winbind package to: samba-winbind-3.6.23-43.el6_9.x86_64.rpm Issue is not seen. Issue seen with these packages: # rpm -qa|grep samba samba-winbind-3.6.23-45.el6_9.x86_64 samba4-libs-4.2.10-11.el6_9.x86_64 samba-common-3.6.23-45.el6_9.x86_64 samba-client-3.6.23-45.el6_9.x86_64 samba-winbind-clients-3.6.23-45.el6_9.x86_64 No Issues with these packages: libsmbclient-3.6.23-43.el6_9.x86_64.rpm samba4-libs-4.2.10-10.el6_9.x86_64.rpm samba-client-3.6.23-43.el6_9.x86_64.rpm samba-common-3.6.23-43.el6_9.x86_64.rpm samba-winbind-3.6.23-43.el6_9.x86_64.rpm samba-winbind-clients-3.6.23-43.el6_9.x86_64.rpm Other Info: 1. selinux is disabled. 2. Issue persists even if no AD user logs into samba server. 3. Has winbind offline logon = false in smb.conf 4. Issue reappears after a while of restarting the winbind service no need for user to login. 5. Customer is ready to test the test-rpms. Version-Release number of selected component (if applicable): samba-winbind-3.6.23-45.el6_9.x86_64 How reproducible: All times in customer env. Steps to Reproduce: 1. Set up samba-6.5 authenticating to AD using winbind 2. winbind package should be samba-winbind-3.6.23-45. 3. start winbind and try connecting to one of the share. 4. Only connection from only 1 user also shows winbind consuming lot of memory. Actual results: winbind memory leak and consumes resources Expected results: winbind should not leak memory and consume resources Additional info: