Description of problem:

We have many customers with huge bricks as big as 200+TBs. In case they want to perform a formatting any one of their replica brick, Healing takes huge time. Though the customer has 20Gbps link between the nodes healing was only able to utilize the network less than 1Gbps. Increasing shd-max threads to 64 and doing multiple client side heal also was not helping. 

Version-Release number of selected component (if applicable):

RHGS 3.1.3

How reproducible:

Always

Steps to Reproduce:
1. Create huge bricks with 50TB+ and create a replica 2 or 3. 
2. Do a formatting of one of the replica brick
3. Add it back and start healing 

Actual results:

Healing takes a very long time, not utilising the hardware capabilities to the max

Expected results:

Healing should be faster and utilize the available network capabilities properly

Additional info: