Bug 1419438

Summary: gluster volume splitbrain info needs to display output of each brick in a stream fashion instead of buffering and dumping at the end
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Nag Pavan Chilakam <nchilaka>
Component: replicateAssignee: Karthik U S <ksubrahm>
Status: CLOSED ERRATA QA Contact: Vijay Avuthu <vavuthu>
Severity: high Docs Contact:
Priority: high    
Version: rhgs-3.2CC: amukherj, ravishankar, rhinduja, rhs-bugs, sheggodu, storage-qa-internal
Target Milestone: ---Keywords: ZStream
Target Release: RHGS 3.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: glusterfs-3.12.2-2 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1506104 (view as bug list) Environment:
Last Closed: 2018-09-04 06:32:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1506104    
Bug Blocks: 1503134    

Description Nag Pavan Chilakam 2017-02-06 07:30:56 UTC 
Description of problem:
=========================
When we issue a heal info, we get the o/p in a continuous stream like fashion.
Say we have 4x2 volume, the the o/p is stream one brick after another

However when we issue a heal info split-brain, the o/p is not streamed as is received but is dumped at the end for all the bricks together.

There are two problems associated with that:
1)This gives a perception that the heal info split-brain is hung
2)we would be simply consuming memory momentarily without releasing till the end of the command(as such no mem leak), but in an extreme case can hypothetically cause a OOM kill(just think of a case where there are lakhs of splitbrain entries in a multibrick volume, and caching all these ) 

Note: I have reproduced it when there are splitbrain entries and even without splitbrain entries

Version-Release number of selected component (if applicable):
==========
3.8.4-13

How reproducible:
==========
always



Steps to Reproduce:
1.create a 2x2 volume
2.create many splibtrain files, say gfid splitbrains on all bricks
3.now issue heal info split-brain



Actual results:
===========

all the o/p is dumped at one shot

Expected results:
===========
keep streaming just as in heal info instead of dumping all at end

Additional info:
Comment 4 Ravishankar N 2017-10-05 04:24:03 UTC 
(In reply to nchilaka from comment #0)

> 
> How reproducible:
> ==========
> always
> 
> 
> 
> Steps to Reproduce:
> 1.create a 2x2 volume
> 2.create many splibtrain files, say gfid splitbrains on all bricks
> 3.now issue heal info split-brain

Karthik, could you take a look at this please? As you know, in glfs-heal.c, all output to stdout is done using printf. I'm wondering if we need to call fflush or if there is some other bug.
Comment 5 Karthik U S 2017-10-05 05:09:54 UTC 
Sure Ravi. Will take a look.
Comment 6 Karthik U S 2017-10-25 07:30:24 UTC 
Upstream patch: https://review.gluster.org/#/c/18570/
Comment 10 Vijay Avuthu 2018-04-18 10:12:26 UTC 
Update:
=========

Build Used : glusterfs-3.12.2-7.el7rhgs.x86_64

Scenario:

1) create 1 * 2 replicate volume
2) create 10K data split-brain files
3) issue heal info split-brain

with this patch , we can see split-brain files are streaming to console

# gluster vol heal 12 info split-brain
Brick 10.70.35.61:/bricks/brick1/b0
/file_140
/file_1
/file_2
/file_3
/file_4
/file_5

Changing status to Verified.
Comment 11 errata-xmlrpc 2018-09-04 06:32:03 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2607