Created attachment 1591664 [details] The attachment is a memory usage statistic of process 31204. Description of problem: Memroy leak in fuse mount process . Version-Release number of selected component (if applicable): # glusterd --version glusterfs 4.1.9 How reproducible: Creating a Distributed-Disperse(8+3) volume,export a directory through Samba. Steps to Reproduce: 1.Creating a Distributed-Disperse(8+3) volume,export a directory through Samba. 2.Writing files to the cifs mount point on windows Server2012R2. 3.The fuse mount process consumes more and more memory . Actual results: Expected results: Additional info: Volume Name: res Type: Distributed-Disperse Volume ID: 22d8a737-d902-4860-a9cf-5a2eb5641c1e Status: Started Snapshot Count: 0 Number of Bricks: 3 x (8 + 3) = 33 Transport-type: tcp Bricks: Brick1: 192.168.100.11:/export/sdb_res/res Brick2: 192.168.100.12:/export/sdb_res/res Brick3: 192.168.100.13:/export/sdb_res/res Brick4: 192.168.100.11:/export/sdc_res/res Brick5: 192.168.100.12:/export/sdc_res/res Brick6: 192.168.100.13:/export/sdc_res/res Brick7: 192.168.100.11:/export/sdd_res/res Brick8: 192.168.100.12:/export/sdd_res/res Brick9: 192.168.100.13:/export/sdd_res/res Brick10: 192.168.100.11:/export/sde_res/res Brick11: 192.168.100.12:/export/sde_res/res Brick12: 192.168.100.13:/export/sde_res/res Brick13: 192.168.100.11:/export/sdf_res/res Brick14: 192.168.100.12:/export/sdf_res/res Brick15: 192.168.100.13:/export/sdf_res/res Brick16: 192.168.100.11:/export/sdg_res/res Brick17: 192.168.100.12:/export/sdg_res/res Brick18: 192.168.100.13:/export/sdg_res/res Brick19: 192.168.100.11:/export/sdh_res/res Brick20: 192.168.100.12:/export/sdh_res/res Brick21: 192.168.100.13:/export/sdh_res/res Brick22: 192.168.100.11:/export/sdi_res/res Brick23: 192.168.100.12:/export/sdi_res/res Brick24: 192.168.100.13:/export/sdi_res/res Brick25: 192.168.100.11:/export/sdj_res/res Brick26: 192.168.100.12:/export/sdj_res/res Brick27: 192.168.100.13:/export/sdj_res/res Brick28: 192.168.100.11:/export/sdk_res/res Brick29: 192.168.100.12:/export/sdk_res/res Brick30: 192.168.100.13:/export/sdk_res/res Brick31: 192.168.100.11:/export/sdl_res/res Brick32: 192.168.100.12:/export/sdl_res/res Brick33: 192.168.100.13:/export/sdl_res/res Options Reconfigured: server.tcp-user-timeout: 3 client.tcp-user-timeout: 5 network.inode-lru-limit: 200000 performance.nl-cache-timeout: 600 performance.nl-cache: on performance.md-cache-timeout: 600 performance.cache-invalidation: on performance.cache-samba-metadata: on performance.stat-prefetch: on features.cache-invalidation-timeout: 600 features.cache-invalidation: on performance.nfs.io-threads: on performance.nfs.quick-read: on performance.client-io-threads: on network.tcp-window-size: 1048567 performance.cache-refresh-timeout: 4 performance.cache-max-file-size: 128MB performance.rda-cache-limit: 20MB performance.parallel-readdir: on cluster.lookup-optimize: on cluster.heal-timeout: 300 network.ping-timeout: 10 server.event-threads: 11 performance.io-thread-count: 40 performance.read-ahead-page-count: 16 performance.write-behind-window-size: 512MB performance.cache-size: 4GB performance.write-behind: on features.quota-deem-statfs: on features.inode-quota: on features.quota: on transport.address-family: inet nfs.disable: on #cat /etc/samba/smb.conf use sendfile = no [testcifs] path = /mnt/res/file_grp/test writable = yes read only = no guest ok = yes kernel share modes = no posix locking = no map archive = no map hidden = no map read only = no map system = no store dos attributes = yes create mode = 0770 directory mode = 2770 map acl inherit = yes oplocks = yes level2 oplocks = yes dos filemode = no dos filetime resolution = no fake directory create times = no dos filetimes = no csc policy = manual browseable = yes [root@node-2 ~]# ps -ef | grep 31204 root 15320 39831 0 Jul17 pts/0 00:00:00 grep --color=auto 31204 root 21121 7093 0 10:12 pts/3 00:00:00 grep --color=auto 31204 root 31204 1 99 Jul17 ? 1-00:05:53 /usr/sbin/glusterfs --acl --process-name fuse --volfile-server=192.168.100.12 --volfile-id=res /mnt/res [root@node-2 ~]# top top - 10:12:30 up 1 day, 14:47, 4 users, load average: 7.65, 7.55, 8.14 Tasks: 714 total, 1 running, 350 sleeping, 3 stopped, 0 zombie %Cpu(s): 2.4 us, 0.8 sy, 0.0 ni, 96.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 13170401+total, 49424232 free, 76349952 used, 5929832 buff/cache KiB Swap: 4194300 total, 4194300 free, 0 used. 53584984 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 31204 root 20 0 66.013g 0.064t 9904 S 100.0 52.0 1445:55 glusterfs 21215 root 20 0 168572 5200 3892 R 11.8 0.0 0:00.03 top 12579 it 20 0 443260 24080 20220 S 5.9 0.0 36:31.31 smbd 1 root 20 0 192712 7076 3824 S 0.0 0.0 1:21.12 systemd 2 root 20 0 0 0 0 S 0.0 0.0 0:01.07 kthreadd 4 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 kworker/0:0H 6 root 20 0 0 0 0 I 0.0 0.0 0:18.93 kworker/u80:0 The attachment is a memory usage statistic of process 31204.
Have you tried using vfs glusterfs module for connecting to the share via Samba?
Due to another bug (https://lists.gluster.org/pipermail/gluster-users/2019-January/035620.html), we can't use vfs glusterfs module. And this bug has not been fixed up in gfs4.1.9. If we use vfs glusterfs module, file access wil bypass fuse mount .Perhaps this bug can't show up. Do I need to retry this?
(In reply to guolei from comment #2) > Due to another bug > (https://lists.gluster.org/pipermail/gluster-users/2019-January/035620.html), > we can't use vfs glusterfs module. > And this bug has not been fixed up in gfs4.1.9. Please note that the fix for issue mentioned in the above mail thread is from Samba and not from GlusterFS and is available with v4.8.6 onwards. But I don't think v4.8.6 or higher is available with CentOS in case you are on CentOS. > If we use vfs glusterfs module, file access wil bypass fuse mount .Perhaps > this bug can't show up. Yes. FUSE layer is not involved when vfs glusterfs module is being used to connect to glusterfs volumes shares. Still you will have to make sure that proper Samba version is installed. > Do I need to retry this? The other bug is seen with creation/renaming of files/directories at root of the share. Just for the sake of verifying this bug you may try using vfs_glusterfs module avoiding operations at root if you can't get hold of required Samba version.
Do we have an update here?
The other bug is seen with creation/renaming of files/directories at root of the share. Just for the sake of verifying this bug you may try using vfs_glusterfs module avoiding operations at root if you can't get hold of required Samba version. -> I tried to use vfs_glusterfs module (smb4.8.3) and I found the fuse mount process consume litte memory. But the smb process consume much more memory than usual. If you need more info, Let me know. Thanks very much.
Here was the output of "top" command, when I accessed volume via smb using vfs_glusterfs module . Tasks: 721 total, 2 running, 352 sleeping, 0 stopped, 0 zombie %Cpu(s): 9.8 us, 9.2 sy, 0.0 ni, 80.4 id, 0.1 wa, 0.0 hi, 0.5 si, 0.0 st KiB Mem : 98674000 total, 558548 free, 33681528 used, 64433928 buff/cache KiB Swap: 4194300 total, 4194300 free, 0 used. 62794580 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 8843 tom 20 0 16.802g 0.014t 265376 R 91.8 15.0 2380:11 smbd 1721 root 20 0 31.983g 3.419g 16616 S 0.0 3.6 12:45.74 java 4176 root 20 0 4554944 1.149g 7972 S 78.6 1.2 1898:52 glusterfsd 4156 root 20 0 4357068 1.077g 8120 S 61.8 1.1 1858:34 glusterfsd 4182 root 20 0 4223136 1.065g 8104 S 76.6 1.1 1881:06 glusterfsd 4115 root 20 0 4231652 1.032g 8148 S 51.6 1.1 1807:30 glusterfsd 4188 root 20 0 4281684 1.030g 8032 S 61.8 1.1 1802:19 glusterfsd 4122 root 20 0 4223168 1.025g 8268 S 56.6 1.1 1872:42 glusterfsd 4155 root 20 0 4224728 1.017g 8400 S 78.6 1.1 1789:53 glusterfsd 4146 root 20 0 4223168 1.017g 8140 S 56.6 1.1 1868:54 glusterfsd 4131 root 20 0 4228336 1.015g 8196 S 54.3 1.1 1884:54 glusterfsd 4132 root 20 0 4158672 1.015g 8216 S 73.4 1.1 1795:58 glusterfsd
(In reply to guolei from comment #5) > I tried to use vfs_glusterfs module (smb4.8.3) and I found the fuse mount > process consume litte memory. Mostly because you don't have an active connection to the share via FUSE mount since you switched to using vfs_glusterfs. > But the smb process consume much more memory than usual. It will consume more than in the previous situation where FUSE mount was used. Because the entire glusterfs client stack is getting loaded into smbd and acts as a client to glusterfs. You will have to figure out by what quantity memory footprint increased in smbd(when vfs_glusterfs is used) and compare it with memory of "glusterfs" process recorded from when FUSE mount was shared.
what information do you need? How can I provide them?
This bug is moved to https://github.com/gluster/glusterfs/issues/988, and will be tracked there from now on. Visit GitHub issues URL for further details