Bug 1372729

Description of problem:

Given replica 2 serving mainly one big distributed-replicated 5 × 2 volume (backups with lots of operations with small files):

===
Volume Name: backups
Type: Distributed-Replicate
...
Status: Started
Number of Bricks: 5 x 2 = 10
Transport-type: tcp
Bricks:
Brick1: pussy.example.com:/mnt/backups_1/backups
Brick2: heimdall.example.com:/mnt/backups_1/backups
Brick3: pussy.example.com:/mnt/backups_2/backups
Brick4: heimdall.example.com:/mnt/backups_2/backups
Brick5: pussy.example.com:/mnt/backups_3/backups
Brick6: heimdall.example.com:/mnt/backups_3/backups
Brick7: pussy.example.com:/mnt/backups_4/backups
Brick8: heimdall.example.com:/mnt/backups_4/backups
Brick9: pussy.example.com:/mnt/backups_5/backups
Brick10: heimdall.example.com:/mnt/backups_5/backups
Options Reconfigured:
cluster.entry-self-heal: off
cluster.metadata-self-heal: off
cluster.data-self-heal: off
performance.readdir-ahead: on
nfs.disable: on
===

Over the last month we observe constant VSZ usage of bricks processes. After updating to 3.7.15 it seems the issue is still there (please see graphs attached).

Version-Release number of selected component (if applicable):

3.7.14, 3.7.15.

How reproducible:

Always.

Steps to Reproduce:
1. create distributed-replicated 5 × 2 volume;
2. perform lots of operations with small files (like git commit on GlusterFS volume — we backup /etc of many servers to it);
3. ...
4. PROFIT.

Actual results:

VSZ grows slowly.

Expected results:

VSZ should not grow to that values.

Additional info:

Graphs attached represent sum of VSZ values over all glusterfsd processes within one node.

RSS also grows, starting from ~80M at the beginning to ~300M before node upgrade. It is considerably larger value than is observed for volumes without such a workload with millions of files.