Bug 1419438 - gluster volume splitbrain info needs to display output of each brick in a stream fashion instead of buffering and dumping at the end
Summary: gluster volume splitbrain info needs to display output of each brick in a str...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: replicate
Version: rhgs-3.2
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: RHGS 3.4.0
Assignee: Karthik U S
QA Contact: Vijay Avuthu
URL:
Whiteboard:
Depends On: 1506104
Blocks: 1503134
TreeView+ depends on / blocked
 
Reported: 2017-02-06 07:30 UTC by nchilaka
Modified: 2018-09-17 14:41 UTC (History)
6 users (show)

Fixed In Version: glusterfs-3.12.2-2
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1506104 (view as bug list)
Environment:
Last Closed: 2018-09-04 06:32:03 UTC


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2018:2607 None None None 2018-09-04 06:33:46 UTC

Description nchilaka 2017-02-06 07:30:56 UTC
Description of problem:
=========================
When we issue a heal info, we get the o/p in a continuous stream like fashion.
Say we have 4x2 volume, the the o/p is stream one brick after another

However when we issue a heal info split-brain, the o/p is not streamed as is received but is dumped at the end for all the bricks together.

There are two problems associated with that:
1)This gives a perception that the heal info split-brain is hung
2)we would be simply consuming memory momentarily without releasing till the end of the command(as such no mem leak), but in an extreme case can hypothetically cause a OOM kill(just think of a case where there are lakhs of splitbrain entries in a multibrick volume, and caching all these ) 

Note: I have reproduced it when there are splitbrain entries and even without splitbrain entries

Version-Release number of selected component (if applicable):
==========
3.8.4-13

How reproducible:
==========
always



Steps to Reproduce:
1.create a 2x2 volume
2.create many splibtrain files, say gfid splitbrains on all bricks
3.now issue heal info split-brain



Actual results:
===========

all the o/p is dumped at one shot

Expected results:
===========
keep streaming just as in heal info instead of dumping all at end

Additional info:

Comment 4 Ravishankar N 2017-10-05 04:24:03 UTC
(In reply to nchilaka from comment #0)

> 
> How reproducible:
> ==========
> always
> 
> 
> 
> Steps to Reproduce:
> 1.create a 2x2 volume
> 2.create many splibtrain files, say gfid splitbrains on all bricks
> 3.now issue heal info split-brain

Karthik, could you take a look at this please? As you know, in glfs-heal.c, all output to stdout is done using printf. I'm wondering if we need to call fflush or if there is some other bug.

Comment 5 Karthik U S 2017-10-05 05:09:54 UTC
Sure Ravi. Will take a look.

Comment 6 Karthik U S 2017-10-25 07:30:24 UTC
Upstream patch: https://review.gluster.org/#/c/18570/

Comment 10 Vijay Avuthu 2018-04-18 10:12:26 UTC
Update:
=========

Build Used : glusterfs-3.12.2-7.el7rhgs.x86_64

Scenario:

1) create 1 * 2 replicate volume
2) create 10K data split-brain files
3) issue heal info split-brain

with this patch , we can see split-brain files are streaming to console

# gluster vol heal 12 info split-brain
Brick 10.70.35.61:/bricks/brick1/b0
/file_140
/file_1
/file_2
/file_3
/file_4
/file_5

Changing status to Verified.

Comment 11 errata-xmlrpc 2018-09-04 06:32:03 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2607


Note You need to log in before you can comment on or make changes to this bug.