Bug 1164906 - Geo-replication changelog crawl causes cpu spike - creates lot of stale changelog files
Summary: Geo-replication changelog crawl causes cpu spike - creates lot of stale chang...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Gluster Storage
Classification: Red Hat Storage
Component: geo-replication
Version: rhgs-3.0
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ---
: RHGS 3.0.4
Assignee: Aravinda VK
QA Contact: shilpa
URL:
Whiteboard:
Depends On: 1169331
Blocks: 1182947
TreeView+ depends on / blocked
 
Reported: 2014-11-17 20:33 UTC by Harshavardhana
Modified: 2018-12-09 19:11 UTC (History)
10 users (show)

Fixed In Version: glusterfs-3.6.0.46-1
Doc Type: Bug Fix
Doc Text:
Previously, geo-replication was not cleaning up processed Changelog files and the inode space would fill the brick. With this fix, Changelog files are archived after processing and hence these files will not consume inodes.
Clone Of:
: 1188968 (view as bug list)
Environment:
Last Closed: 2015-03-26 06:34:57 UTC
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2015:0682 0 normal SHIPPED_LIVE Red Hat Storage 3.0 enhancement and bug fix update #4 2015-03-26 10:32:55 UTC

Comment 4 Aravinda VK 2015-01-15 10:30:02 UTC 
Upstream patch sent for review. http://review.gluster.org/#/c/9453/
Comment 5 Kotresh HR 2015-02-17 08:47:17 UTC 
The upstream patch: http://review.gluster.org/#/c/9453/
is the one that archives changelogs in working directory and avoids
creation of zero byte xsync changelogs which addresses this bug.
Once, it is merged upstream, backporting it should fix this bug.
The corresponding upstream bug is
https://bugzilla.redhat.com/show_bug.cgi?id=1169331


The upstream patch: http://review.gluster.org/#/c/9572/
is to avoid creation of zero byte changelog files at backend
which is different.
Comment 6 Aravinda VK 2015-02-20 04:29:47 UTC 
Downstream patch sent https://code.engineering.redhat.com/gerrit/#/c/42344/
Comment 7 shilpa 2015-02-25 11:09:28 UTC 
Tested and verified the two patches mentioned in comment#5 on version glusterfs-3.6.0.46-1.el6rhs.x86_64. 

After the changelogs are processed an archive.tar is created in processed directory. 

In my testbed: 

Working directory:  /var/lib/misc/glusterfsd/master/ssh%3%2F%2Froot%4010.x.x.x%3Agluster%3A%2F%2F127.0.0.1%3Aslave

We would see an archive file for the processed changelog. Validated the entries in it:

./6ba51664ad363afecedbc65396e2c80c/.processed/archive_201502.tar

Validated the same for xsync and history as well. 

./a8923163236aecc5b19cb96e2e94f8f7/xsync/archive_201502.tar
./6ba51664ad363afecedbc65396e2c80c/.history/.processed/archive_201502.tar

Note: I have not tested the CPU spikes mentioned in the test case as we are not really sure what was the customer's workload at that time.
Comment 9 Bhavana 2015-03-23 19:33:33 UTC 
Hi Aravinda,

The doc text is updated. review the same and sign off if it looks ok.
Comment 10 Aravinda VK 2015-03-24 05:57:13 UTC 
(In reply to Bhavana from comment #9)
> Hi Aravinda,
> 
> The doc text is updated. review the same and sign off if it looks ok.

looks good to me.
Comment 12 errata-xmlrpc 2015-03-26 06:34:57 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-0682.html
Note You need to log in before you can comment on or make changes to this bug.