RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1513877 - Regression: winbind leaks memory after upgrade
Summary: Regression: winbind leaks memory after upgrade
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: samba
Version: 6.5
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: rc
: ---
Assignee: Andreas Schneider
QA Contact: Robin Hack
URL:
Whiteboard:
: 1558933 (view as bug list)
Depends On:
Blocks: 1504542
TreeView+ depends on / blocked
 
Reported: 2017-11-16 08:09 UTC by amitkuma
Modified: 2021-12-10 15:24 UTC (History)
18 users (show)

Fixed In Version: samba-3.6.23-51.el6
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-06-19 05:08:55 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
valgrind output (3.30 MB, application/x-gzip)
2017-11-16 08:09 UTC, amitkuma 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2018:1860 0 None None None 2018-06-19 05:10:08 UTC

Description amitkuma 2017-11-16 08:09:26 UTC 
Created attachment 1353272 [details]
valgrind output

Description of problem:
After the winbind package upgraded from samba-winbind-3.6.23-43.el6_9 to samba-winbind-3.6.23-45.el6_9, Customer noticed it consumes huge memory and then oom-killer is invoked and kills the process.

$ dmesg|grep -i killed
Killed process pid, UID 0, (winbindd) total-vm:206912kB, anon-rss:572kB, file-rss:124kB
Killed process pid3, UID 0, (winbindd) total-vm:204672kB, anon-rss:296kB, file-rss:788kB
Killed process pid4, UID 0, (winbindd) total-vm:204836kB, anon-rss:292kB, file-rss:812kB
Killed process pid7, UID 0, (winbindd) total-vm:204836kB, anon-rss:292kB, file-rss:704kB

If customer downgrades winbind package to:
samba-winbind-3.6.23-43.el6_9.x86_64.rpm
Issue is not seen.

Issue seen with these packages:
# rpm -qa|grep samba
samba-winbind-3.6.23-45.el6_9.x86_64
samba4-libs-4.2.10-11.el6_9.x86_64
samba-common-3.6.23-45.el6_9.x86_64
samba-client-3.6.23-45.el6_9.x86_64
samba-winbind-clients-3.6.23-45.el6_9.x86_64

No Issues with these packages:
libsmbclient-3.6.23-43.el6_9.x86_64.rpm
samba4-libs-4.2.10-10.el6_9.x86_64.rpm
samba-client-3.6.23-43.el6_9.x86_64.rpm
samba-common-3.6.23-43.el6_9.x86_64.rpm
samba-winbind-3.6.23-43.el6_9.x86_64.rpm
samba-winbind-clients-3.6.23-43.el6_9.x86_64.rpm

Other Info:
1. selinux is disabled.
2. Issue persists even if no AD user logs into samba server.
3. Has winbind offline logon = false in smb.conf
4. Issue reappears after a while of restarting the winbind service no need for user to login.
5. Customer is ready to test the test-rpms.

Version-Release number of selected component (if applicable):
samba-winbind-3.6.23-45.el6_9.x86_64

How reproducible:
All times in customer env.

Steps to Reproduce:
1. Set up samba-6.5 authenticating to AD using winbind
2. winbind package should be samba-winbind-3.6.23-45.
3. start winbind and try connecting to one of the share.
4. Only connection from only 1 user also shows winbind consuming lot of memory.

Actual results:
winbind memory leak and consumes resources

Expected results:
winbind should not leak memory and consume resources

Additional info:
Comment 2 amitkuma 2017-11-16 08:31:06 UTC 
Customer provided information.
Issue is also present on samba-winbind-3.6.23-44
Comment 3 amitkuma 2017-11-17 06:14:56 UTC 
Hello,

Another Customer(Cargill Inc) using samba-3.6.23-45.el6_9 found winbind leak Issues.

# tail -f /var/log/samba/winbindd.log
[2017/11/16 12:30:45.012275,  0] winbindd/winbindd_dual.c:1392(fork_domain_child)
  Could not fork: Cannot allocate memory
[2017/11/16 12:30:45.015812,  0] winbindd/winbindd_dual.c:1392(fork_domain_child)
  Could not fork: Cannot allocate memory
[2017/11/16 12:30:45.021137,  0] winbindd/winbindd_dual.c:1392(fork_domain_child)
  Could not fork: Cannot allocate memory
[2017/11/16 12:32:01.927146,  0] param/loadparm.c:8056(lp_do_parameter)
  Global parameter unix extensions found in service section!
[2017/11/16 12:32:01.937565,  0] winbindd/winbindd_cache.c:3204(initialize_winbindd_cache)
  initialize_winbindd_cache: clearing cache and re-creating with version number 2

# tail -f /var/log/messages
Nov 16 12:30:50 XXXX kernel: [20260]     0 20260     1026       17   2       0             0 tail
Nov 16 12:30:50 XXXX kernel: [30144]     0 30144   102022      262   3       0             0 LogMonitoring
Nov 16 12:30:50 XXXX kernel: Out of memory: Kill process 15343 (winbindd) score 637 or sacrifice child
Nov 16 12:30:50 XXXX kernel: Killed process 15343, UID 0, (winbindd) total-vm:39083488kB, anon-rss:22851136kB, file-rss:36kB
Nov 16 12:32:01 XXXX winbindd[9283]: [2017/11/16 12:32:01.927146,  0] param/loadparm.c:8056(lp_do_parameter)
Nov 16 12:32:01 XXXX winbindd[9283]:   Global parameter unix extensions found in service section!
Nov 16 12:32:01 XXXX winbindd[9285]: [2017/11/16 12:32:01.937565,  0] winbindd/winbindd_cache.c:3204(initialize_winbindd_cache)
Nov 16 12:32:01 XXXX winbindd[9285]:   initialize_winbindd_cache: clearing cache and re-creating with version number 2

valgrind Output is taken after running:
# /usr/bin/valgrind --trace-children=yes --show-reachable=yes --track-origins=yes --read-var-info=yes --tool=memcheck --leak-check=full --num-callers=50 -v --log-file=/tmp/valgrind.out /usr/sbin/winbindd -F -S

Thanks
Amit
Comment 4 Andreas Schneider 2017-11-21 08:42:35 UTC 
Please always install the 'samba-debuginfo' package before you run valgrind!


USE: valgrind --tool=memcheck -v --num-callers=20 --track-origins=yes --trace-children=yes


DO NOT ADD: --leak-check nor --show-reachables
Comment 34 Andreas Schneider 2018-01-23 15:12:30 UTC 
It looked like from previous information that the issue is in the main winbindd process (see 'ps afx'). Assuming that this piece of information is correct we will need the following steps:

1. Upgrade the samba packages that had the issue.
2. Install samba-debuginfo libtevent-debuginfo
3. Enable samba debug logs

service winbind stop; service smb stop; service nmb stop;

     log level = 10
     debug pid = true
     max log size = 0

rm -f /var/log/samba/*
service smb start; service nmb start

4. valgrind --tool=memcheck -v --num-callers=20 --track-origins=yes --leak-check=full /usr/sbin/winbindd  and wait for the issue to happen (the memory of winbindd starts growing and growing)

5. Provide samba and valgrind logs including the output of 'wbinfo --trusted-domains --verbose"'
Comment 36 Andreas Schneider 2018-02-08 11:20:29 UTC 
Where can I find the logs?
Comment 43 Andreas Schneider 2018-02-20 12:48:09 UTC 
Could I get the coredump please?
Comment 59 Andreas Schneider 2018-03-21 16:36:10 UTC 
*** Bug 1558933 has been marked as a duplicate of this bug. ***
Comment 71 errata-xmlrpc 2018-06-19 05:08:55 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:1860
Note You need to log in before you can comment on or make changes to this bug.