Bug 1461543 - [Ganesha] : Ganesha occupies increased memory (RES size) even when all the files are deleted from the mount
[Ganesha] : Ganesha occupies increased memory (RES size) even when all the fi...
Status: CLOSED ERRATA
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: nfs-ganesha (Show other bugs)
3.3
x86_64 Linux
unspecified Severity high
: ---
: RHGS 3.3.0
Assigned To: Kaleb KEITHLEY
Ambarish
3.3.0-devel-freeze-exception
:
Depends On: 1477668 1481392
Blocks: 1417151
  Show dependency treegraph
 
Reported: 2017-06-14 14:11 EDT by Ambarish
Modified: 2017-09-21 00:59 EDT (History)
18 users (show)

See Also:
Fixed In Version: glusterfs-3.8.4-38
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1481392 1481925 1481951 (view as bug list)
Environment:
Last Closed: 2017-09-21 00:59:42 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Summary from massif output (27.25 KB, text/plain)
2017-06-27 19:02 EDT, Frank Filz
no flags Details
Massif snapshot (24.94 KB, text/plain)
2017-06-28 14:31 EDT, Daniel Gryniewicz
no flags Details
massif output on Dev setup,post rm (38.32 KB, text/plain)
2017-06-28 14:39 EDT, Ambarish
no flags Details
massif_snaphost_idle_setup_post_io (35.29 KB, text/plain)
2017-06-29 08:11 EDT, Soumya Koduri
no flags Details
mem-pool: mem_put() returns to current thread pool (1.78 KB, patch)
2017-07-19 09:42 EDT, Niels de Vos
no flags Details | Diff
Test case that didn't reproduce the issue (4.03 KB, patch)
2017-07-28 08:41 EDT, Poornima G
no flags Details | Diff

  None (edit)
Description Ambarish 2017-06-14 14:11:22 EDT
Description of problem:
----------------------

2 Node Ganesha cluster.

Ran Bonnie from three v4 mounts

** Before Bonnie was run **:

Tasks: 314 total,   1 running, 313 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.6 us,  0.3 sy,  0.0 ni, 99.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 49279520 total, 38703092 free,  1535844 used,  9040584 buff/cache
KiB Swap: 24772604 total, 24697744 free,    74860 used. 46968336 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                            
18825 root      20   0 2227148 118932   3808 S  10.0  0.2  25:24.27 glusterfsd                                                         
15816 root      20   0 6264984  17220   5148 S   8.0  0.0   0:07.66 ganesha.nfsd                                                       
18791 root      20   0 2560808 494140   3828 S   7.3  1.0 137:42.27 glusterfsd                                                         
 6561 root      rt   0  191716  93452  70732 S   1.0  0.2   2:55.14 corosync                                                           
   10 root      20   0       0      0      0 S   0.3  0.0   0:53.10 rcu_sched                                                          
 3489 root       0 -20       0      0      0 S   0.3  0.0   0:00.07 kworker/0:1H                                                       
 6569 haclust+  20   0  110104  17868  10272 S   0.3  0.0   0:47.97 cib                                                                
10044 root      20   0       0      0      0 S   0.3  0.0   0:07.75 kworker/u64:1                                                      
17126 root      20   0       0      0      0 S   0.3  0.0   0:11.63 xfsaild/dm-7                                                       
17186 root      20   0  158004   2512   1544 R   0.3  0.0   0:00.11 top                                                                
    1 root      20   0  192428   3416   2168 S   0.0  0.0   0:06.62 systemd                                                            
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.02 kthreadd                                                           
    3 root      20   0       0      0      0 S   0.0  0.0   0:01.01 ksoftirqd/0                                                        
    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H                                                       
    8 root      rt   0       0      0      0 S   0.0  0.0   0:00.43 migration/0                                                        
    9 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_bh 



**After Bonnie was run + rm to clean up mount** :

top - 14:00:35 up 10:46,  1 user,  load average: 0.13, 0.17, 0.62
Tasks: 339 total,   2 running, 337 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.3 us,  0.1 sy,  0.0 ni, 99.6 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 49279520 total, 31473508 free,  8127248 used,  9678764 buff/cache
KiB Swap: 24772604 total, 24680888 free,    91716 used. 40454128 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                            
26055 root      20   0 18.099g 5.520g   5204 S   7.6 11.7 136:32.63 ganesha.nfsd                                                       
 6561 root      rt   0  191720  93456  70732 S   1.0  0.2   5:47.31 corosync                                                           
18384 root      20   0       0      0      0 S   0.3  0.0   0:00.21 kworker/u64:2                                                      
22926 root      20   0  158008   2540   1544 R   0.3  0.0   0:00.17 top                                                                
    1 root      20   0  192428   3356   1952 S   0.0  0.0   0:09.04 systemd                                                            
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.03 kthreadd                                                           
    3 root      20   0       0      0      0 S   0.0  0.0   0:01.78 ksoftirqd/0                                                        
    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H                                                       
    8 root      rt   0       0      0      0 S   0.0  0.0   0:00.54 migration/0                                                        
    9 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_bh                                                             
   10 root      20   0       0      0      0 S   0.0  0.0   1:07.59 rcu_sched                                                          
   11 root      rt   0       0      0      0 S   0.0  0.0   0:00.13 watchdog/0                                                         
   12 root      rt   0       0      0      0 S   0.0  0.0   0:00.12 watchdog/1                                                         
   13 root      rt   0       0      0      0 S   0.0  0.0   0:00.63 migration/1                                                        
   14 root      20   0       0      0      0 S   0.0  0.0   0:01.97 ksoftirqd/1                                                        
   16 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/1:0H                                                       
   18 root      rt   0       0      0      0 S   0.0  0.0   0:00.10 watchdog/2                                                         
   19 root      rt   0       0      0      0 S   0.0  0.0   0:00.54 migration/2                                                        
   20 root      20   0       0      0      0 S   0.0  0.0   0:01.51 ksoftirqd/2                                                        
   22 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/2:0H                                                       
   23 root      rt   0       0      0      0 S   0.0  0.0   0:00.11 watchdog/ 


Since the memory is not being released , I am suspecting a mem leak here.

Version-Release number of selected component (if applicable):
-------------------------------------------------------------

nfs-ganesha-2.4.4-8.el7rhgs.x86_64
glusterfs-ganesha-3.8.4-25.el7rhgs.x86_64


[root@gqas007 glusterfs]# rpm -qa|grep  libnti
libntirpc-devel-1.4.3-4.el7rhgs.x86_64
libntirpc-1.4.3-4.el7rhgs.x86_64
[root@gqas007 glusterfs]# 


[root@gqas007 glusterfs]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.4 Beta (Maipo)
[root@gqas007 glusterfs]#

How reproducible:
-----------------

2/2,6G is the maximum I have seen.4.5G+ is fairly consistent.


Additional info:
----------------

Volume Name: testvol
Type: Distributed-Replicate
Volume ID: 3b04b36a-1837-48e8-b437-fbc091b2f992
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: gqas007.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick0
Brick2: gqas009.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick1
Brick3: gqas007.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick2
Brick4: gqas009.sbu.lab.eng.bos.redhat.com:/bricks/testvol_brick3
Options Reconfigured:
ganesha.enable: on
features.cache-invalidation: on
server.allow-insecure: on
performance.stat-prefetch: off
transport.address-family: inet
nfs.disable: on
nfs-ganesha: enable
cluster.enable-shared-storage: enable
[root@gqas009 bricks]#
Comment 7 Ambarish 2017-06-15 07:55:41 EDT
Lost qa_ack in collision.
Reset now.
Comment 11 Daniel Gryniewicz 2017-06-15 13:14:20 EDT
So, most of those leaks are in gluster; however, it looks like we are leaking MDCACHE entries, but only 1.5MB worth, so it shouldn't be causing issues...

Total leak is only 150MB, but total unleaked is only 58MB, so it's leaking 3x.
Comment 13 Soumya Koduri 2017-06-15 13:46:18 EDT
Could you please attach the statedump file of ganesha/gfapi  process.
Comment 15 Soumya Koduri 2017-06-15 14:09:45 EDT
@Ambarish,

But the ganesha process PIDs mentioned in the bug description (comment#1) are 15816 & 26055. Are you sure the statedump provided above was taken post running the tests (i.e, when ganesha process is consuming lot of memory)? I do not see much memory being consumed in the gfapi/gluster stack from the statedump provided.
Comment 16 Ambarish 2017-06-15 14:14:48 EDT
Sorry,I did not notice that.

I'll update this with one of the reproducers.
Comment 17 Soumya Koduri 2017-06-15 14:17:29 EDT
if you are reproducing the issue, could you please collect statedump at regular intervals..
* couple of them while running bonnie, &
* before and after removal of mount contents. Thanks!
Comment 19 Frank Filz 2017-06-21 14:19:17 EDT
In order to chase down any contribution of Ganesha to the huge memory footprint, I think it would help to have availability of a cluster that can reproduce the issue along with instructions on how to run the scenario.

I really want to get output from massif under the suspect workload so I can see just where all the memory is being used.
Comment 22 Frank Filz 2017-06-27 19:02 EDT
Created attachment 1292484 [details]
Summary from massif output

I have done an analysis of massif output. At it's peak, it doesn't look like 6 GB of memory is used on the heap, in fact it only shows 172 MB of memory allocated from the heap. That doesn't account for stacks and such, so I don't know if that might be part of the 6 GB of memory, or exactly what contribute.

The main things I see are:

1. hashtable uses a LOT of memory all the time. This is planned to be fixed for 2.6.

2. Gluster is the next biggest use of memory

3. mdcache uses a bunch of memory, however, this memory is all released. Planned for early 2.6 and backported to 2.5 stable is memory management for the dirent cache to limit the size it can attain.

The issue I think we might have been having before is that massif doesn't produce its output until the process is terminated. By terminating the process, I was able to get the output summarized in this attachment.
Comment 23 Daniel Gryniewicz 2017-06-28 11:26:06 EDT
I believe this is not a bug.  Here's my reasoning.

1. VSS (Virtual memory used) does not equate to memory used.  Lots of things (most common example is memory mapped files) can use virtual address space without using actual RAM.

2. In the listed output, the RES (actual memory used) is ~170 MB.

3. I've reproduced this on my cluster, seeing high VSS and normal (<200 MB) RES.  I've run this under Massif, and verified that the actual heap use is the same as the RES.

Therefore, I conclude Ganesha is *not* using lots of RAM in these tests.

(Also note that the dirents and obj cache don't even show up in the Massif output, they're so small)
Comment 24 Daniel Gryniewicz 2017-06-28 11:31:50 EDT
I take it back, I missed that one of the RES was huge.  So there is some issue somewhere.
Comment 25 Matt Benjamin (redhat) 2017-06-28 11:42:40 EDT
(In reply to Daniel Gryniewicz from comment #23)
> I believe this is not a bug.  Here's my reasoning.
> 
> 1. VSS (Virtual memory used) does not equate to memory used.  Lots of things
> (most common example is memory mapped files) can use virtual address space
> without using actual RAM.
> 
> 2. In the listed output, the RES (actual memory used) is ~170 MB.
> 
> 3. I've reproduced this on my cluster, seeing high VSS and normal (<200 MB)
> RES.  I've run this under Massif, and verified that the actual heap use is
> the same as the RES.
> 
> Therefore, I conclude Ganesha is *not* using lots of RAM in these tests.
> 
> (Also note that the dirents and obj cache don't even show up in the Massif
> output, they're so small)

The above considerations are valid, nevertheless, I think the memory utilization should be accounted for, as I at least don't think I have a good explanation why we would have anything like 6G of mappings in use.

Matt
Comment 26 Daniel Gryniewicz 2017-06-28 14:30:45 EDT
Okay, I got a snapshot from a run using 2.3G of RES.  Here's the highlights, and I'll attach the full output:


->53.65% (644,874,245B) 0x8F8B5C8: __gf_calloc (mem-pool.c:117)

The bulk is from GFAPI calls.

->20.55% (247,037,160B) 0x8B13B19: __new_var_obj_p (__libobj.c:36)
| ->16.32% (196,158,024B) 0x8B11455: __acl_create_entry_obj (acl_create_entry.c:31)
| | ->16.32% (196,158,024B) 0x8B11496: acl_create_entry (acl_create_entry.c:67)
| ->04.23% (50,879,136B) 0x8B124C6: __acl_init_obj (acl_init.c:28)
|   ->04.23% (50,879,136B) 0x8B1254B: acl_init (acl_init.c:66)

250 MB is from ACLs from FSAL_GLUSTER.  I think these may be leaked, and I'm trying a patch out to see.

->06.73% (80,877,576B) 0x5201DB: gsh_calloc__ (abstract_mem.h:145)
| ->06.73% (80,877,576B) 0x52025E: pool_alloc__ (abstract_mem.h:395)
|   ->06.73% (80,877,576B) 0x5241DB: alloc_cache_entry (mdcache_lru.c:1277)

80 MB from the Ganesha handle cache

->04.06% (48,790,984B) 0x4E0693: gsh_calloc__ (abstract_mem.h:145)
| ->04.06% (48,757,296B) 0x4E1086: hashtable_init (hashtable.c:341)

The standard 48M from Ganesha hash tables

So, the one that may be Ganesha's fault and fixable is the ACLs.
Comment 27 Daniel Gryniewicz 2017-06-28 14:31 EDT
Created attachment 1292689 [details]
Massif snapshot
Comment 28 Ambarish 2017-06-28 14:39 EDT
Created attachment 1292691 [details]
massif output on Dev setup,post rm

I retried this on Dev setup with Frank and Dan(thanks guys!).

Ganesha took ~1.25G of memory post bonnie and subsequent rm/cleanup from mount.

Attaching massif o/p to the bz.
Comment 29 Daniel Gryniewicz 2017-06-28 14:43:42 EDT
The final output from massif does not show the ACLs, so they're not being leaked, they just take up a lot of memory.
Comment 31 Soumya Koduri 2017-06-29 08:11 EDT
Created attachment 1292855 [details]
massif_snaphost_idle_setup_post_io
Comment 49 Niels de Vos 2017-07-17 07:34:21 EDT
I've been looking at libglusterfs/src/mem-pool.c for cleaning up when glfs_fini() is called. This cleanup is not available yet, patches are being worked on and show up here: https://review.gluster.org/#/q/topic:bug-1470170

That means I have gained some understanding in how the new mem-pools work, but I have not focussed much on the details. So, take the below with some gain of salt, it may not be correct. Writing this while reading the sources.

TL;DR: I don't see any memory leaks in mem-pools itself. It may take up to 2x 30 seconds before memory returned with mem_put() is released.

Because the test-build (where the pool_sweeper thread gets enabled for libgfapi) showed reduced memory usage, we know that at least some of the functionality is working as expected.

If this is a problem with the new mem-pool implementation, it should not occur when running these tests with a glusterfs build that has --disable-mempool in its ./configure options.

Unfortunately the new mem-pools do not offer statedump features. It is difficult to debug problems in case mem_get() allocations are not free'd. (This is also the case when building with --disable-mempool.)


-- %< -- more notes about the structure of the different structures and lists --
It's really great that doc/developer-guide/datastructure-mem-pool.md was not updated with the new model :-/

There are per-thread-pools, each per_thread_pool has two arrays[NPOOLS=14] of lists (hot_list and cold_list). 

libglusterfs/src/mem-pools.c:
374 #define POOL_SMALLEST   7       /* i.e. 128 */
375 #define POOL_LARGEST    20      /* i.e. 1048576 */
376 #define NPOOLS          (POOL_LARGEST - POOL_SMALLEST + 1)

A mem_get() rounds-up the allocation to the next power-of-2 size and places that allocation (+mem_header) in the per_thread_pool_list (which links to a per_thread_pool mentioned above).

A mem_put() reads the mem_header, and uses the size of the allocation to place it on the associated per_thread_pool_list.hot_list.

At one point (every 30 secs) the pool_sweeper thread goes through all the per_thread_pool_list's.

- If that per_thread_pool_list belonged to a thread that exited, the 
  per_thread_pool_list will have been marked with ->poison. This causes 
  collect_garbage() to move the per_thread_pool_list to death_row (a 
  pool_sweeper state)

- In addition to cleaning up after threads exited, collect_garbage() marks the 
  per_thread_pool_list.cold_list's for removal in pool_sweeper. Each 
  per_thread_pool_list.hot_list is renamed to cold_list (and will be cleaned up
  on the next run of pool_sweeper.

- In pool_sweeper, all per_thread_pool_list's in the death_row will be free'd 
  (everything in the hot_list and _cold_list). The per_thread_pool_list itself 
  moves to the pool_free_threads list where it can be used for a next new 
  thread.

- At last, in pool_sweeper, all cold_lists that were marked as garbage will be
  free'd.
Comment 52 Niels de Vos 2017-07-19 09:42 EDT
Created attachment 1301119 [details]
mem-pool: mem_put() returns to current thread pool

There is a possible problem that could cause memory leaks in the case where threads get started and stopped a lot. I do not know if this is the case with NFS-Ganesha though.

- a mem_get() sets up a mem_header that points to the mem-pool of the current 
  thread
- if the thread exits, the mem-pool of the thread is cleaned (hot_list, 
  cold_list, not active allocations) and moved to pool_free_list
- mem_put() takes the pointer from the mem_header and moves the allocation from 
  the active state to the hot_list for re-use
- inactive threads (listed in pool_free_list) are not cleaned by the pool_sweeper
- the mem_put()'d allocation will linger around indefinitely

This patch moves the allocation to the mem-pool of the currently running thread. This prevents growing the hot_list of inactive per_thread_pool_t structures, so no additional cleaning is needed.

As noted in the commit message of the patch, this theory needs to be proven and further polishing is needed as well.
Comment 57 Poornima G 2017-07-28 08:39:44 EDT
So, since i was told that normal dd operation reproduces the memory leak, took some time to write a gfapi application. What it does is spawn specified number of threads and issue glfs_pwrite on each of those threads and create a big file.

When i ran with 100 threads, each writing 200MB of data, and hence creating on large 20GB file, my observation were:
1. When run on RHGS 3.3: RES memory goes to 10G while the IO is going on, but the moment threads are done with the IO, the memory falls down to as low as 300M.
2. When run on gluster master branch, RES memory didn't ever cross few 100 MBs, hence, definitely there are patches upstream (may be Kalebs gfapi sweeper patch), that needs to be pulled in any case.

Conclusion:
I think its a good idea to try with master build and Ganesha test to check if the issue is already fixed.

2. Its mostly not any simple large file write that causes the leak, as the above test case showed no leaks.


Attaching the test case.
Comment 58 Poornima G 2017-07-28 08:41 EDT
Created attachment 1305946 [details]
Test case that didn't reproduce the issue
Comment 59 Matt Benjamin (redhat) 2017-07-28 08:59:19 EDT
(In reply to Poornima G from comment #57)
> So, since i was told that normal dd operation reproduces the memory leak,
> took some time to write a gfapi application. What it does is spawn specified
> number of threads and issue glfs_pwrite on each of those threads and create
> a big file.
> 
> When i ran with 100 threads, each writing 200MB of data, and hence creating
> on large 20GB file, my observation were:
> 1. When run on RHGS 3.3: RES memory goes to 10G while the IO is going on,
> but the moment threads are done with the IO, the memory falls down to as low
> as 300M.
> 2. When run on gluster master branch, RES memory didn't ever cross few 100
> MBs, hence, definitely there are patches upstream (may be Kalebs gfapi
> sweeper patch), that needs to be pulled in any case.
> 
> Conclusion:
> I think its a good idea to try with master build and Ganesha test to check
> if the issue is already fixed.
> 
> 2. Its mostly not any simple large file write that causes the leak, as the
> above test case showed no leaks.
> 
> 
> Attaching the test case.

I think we need results with corresponding builds of libgfapi and nfs-ganesha serving candidate workloads, as a starting point for decision making.

Matt
Comment 61 Soumya Koduri 2017-07-28 13:09:56 EDT
(In reply to Poornima G from comment #57)
> So, since i was told that normal dd operation reproduces the memory leak,
> took some time to write a gfapi application. What it does is spawn specified
> number of threads and issue glfs_pwrite on each of those threads and create
> a big file.
> 
> When i ran with 100 threads, each writing 200MB of data, and hence creating
> on large 20GB file, my observation were:
> 1. When run on RHGS 3.3: RES memory goes to 10G while the IO is going on,
> but the moment threads are done with the IO, the memory falls down to as low
> as 300M.

@Poornima,
Thanks for the testcase. Just an observation from the test-script. I see that we are doing - 


        printf ("\nspawning threads %d", threads_cnt);
        for (i = 0; i < threads_cnt ; i++) {
                thcnt[i] = i;
                pthread_create (&tid[i], NULL, &thread_func, &thcnt[i]);
        }

        for (i = 0; i < threads_cnt ; i++) {
                pthread_join (tid[i], NULL);
        }

        printf ("\ndone writing");
        scanf ("%d", &tmp);

Did you see memory drop right after writes are done i.e, before pthread_join of threads ? Or was the drop observed after pthread_join. If that is the case then  the threads may have exited and hence cleaned up the memory consumed. 

I am sorry I haven't run the testcase but will give it a try...but just wanted to confirm with you if my doubts are valid. Thanks!
Comment 62 Poornima G 2017-07-29 02:11:38 EDT
(In reply to Soumya Koduri from comment #61)
> (In reply to Poornima G from comment #57)
> > So, since i was told that normal dd operation reproduces the memory leak,
> > took some time to write a gfapi application. What it does is spawn specified
> > number of threads and issue glfs_pwrite on each of those threads and create
> > a big file.
> > 
> > When i ran with 100 threads, each writing 200MB of data, and hence creating
> > on large 20GB file, my observation were:
> > 1. When run on RHGS 3.3: RES memory goes to 10G while the IO is going on,
> > but the moment threads are done with the IO, the memory falls down to as low
> > as 300M.
> 
> @Poornima,
> Thanks for the testcase. Just an observation from the test-script. I see
> that we are doing - 
> 
> 
>         printf ("\nspawning threads %d", threads_cnt);
>         for (i = 0; i < threads_cnt ; i++) {
>                 thcnt[i] = i;
>                 pthread_create (&tid[i], NULL, &thread_func, &thcnt[i]);
>         }
> 
>         for (i = 0; i < threads_cnt ; i++) {
>                 pthread_join (tid[i], NULL);
>         }
> 
>         printf ("\ndone writing");
>         scanf ("%d", &tmp);
> 
> Did you see memory drop right after writes are done i.e, before pthread_join
> of threads ? Or was the drop observed after pthread_join. If that is the
> case then  the threads may have exited and hence cleaned up the memory
> consumed. 
> 
> I am sorry I haven't run the testcase but will give it a try...but just
> wanted to confirm with you if my doubts are valid. Thanks!

In each thread i have written a scanf(), so that the thread doesn't join, bad hack but serves the purpose. In the thread_func:
thread_func (void *opaque)
{
        ...
        write 200MB in 1MB chunks
        printf ("\n Done with thread:%d", *((int *)opaque));
scanf ("%d", &j);
        return NULL;
}

The memory dropped, when all the threads were done writing, and were in scanf, and not yet joined.
Comment 63 Soumya Koduri 2017-07-29 04:57:43 EDT
> In each thread i have written a scanf(), so that the thread doesn't join,
> bad hack but serves the purpose. In the thread_func:
> thread_func (void *opaque)
> {
>         ...
>         write 200MB in 1MB chunks
>         printf ("\n Done with thread:%d", *((int *)opaque));
> scanf ("%d", &j);
>         return NULL;
> }
> 
> The memory dropped, when all the threads were done writing, and were in
> scanf, and not yet joined.

Oh okay..thanks for the confirmation!
Comment 77 errata-xmlrpc 2017-09-21 00:59:42 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2774

Note You need to log in before you can comment on or make changes to this bug.