Bug 980822 - Dist-geo-rep : After heavy IO, the geo-rep fails to sync few files, because it fails to get few changelogs. [NEEDINFO]
Dist-geo-rep : After heavy IO, the geo-rep fails to sync few files, because i...
Status: CLOSED WORKSFORME
Product: Red Hat Gluster Storage
Classification: Red Hat
Component: geo-replication (Show other bugs)
2.1
x86_64 Linux
high Severity high
: ---
: ---
Assigned To: Bug Updates Notification Mailing List
amainkar
: ZStream
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-07-03 06:01 EDT by Vijaykumar Koppad
Modified: 2015-03-17 10:42 EDT (History)
10 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-12-24 00:54:10 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
vshankar: needinfo? (vkoppad)


Attachments (Terms of Use)

  None (edit)
Description Vijaykumar Koppad 2013-07-03 06:01:38 EDT
Description of problem: Run heavy IOs, like dbench 50, iozone and bonnie++ parallely on different clients of the master. The geo-rep fails to sync few files because it doesn't get the changelog files to process. This is because pipe between rollover thread and notification thread got full.


Version-Release number of selected component (if applicable): 
glusterfs-3.4.0.12rhs.beta1-1.el6rhs.x86_64

How reproducible: Got hit twice. 


Steps to Reproduce:
1.Create and start a geo-rep session between master( dist-rep(2x2)) and slave.
2.Run heavy IOs on like dbench 50, iozone, and bonnie++ on different clients of the master.
3.Check on the slave whether it completes the syncing.

Actual results:It fails to sync few files.


Expected results:It should sync all the files from master to slave. 


Additional info:

Logs from the bricks from where it failed to sync few files, 

[2013-07-02 02:14:22.834366] E [changelog-helpers.c:161:changelog_rollover_changelog] 0-master-changelog: Failed to send file name to notify thread (reason: Resource temporarily unavailable)
[2013-07-02 02:14:22.834449] E [changelog-helpers.c:272:changelog_handle_change] 0-master-changelog: Problem rolling over changelog(s)
[2013-07-02 02:14:23.275960] E [changelog-helpers.c:281:changelog_handle_change] 0-master-changelog: fsync failed (reason: Bad file descriptor)
[2013-07-02 02:14:23.276023] E [changelog-helpers.c:442:changelog_fsync_thread] 0-master-changelog: failed to inject fsync event
[2013-07-02 02:14:28.281175] E [changelog-helpers.c:281:changelog_handle_change] 0-master-changelog: fsync failed (reason: Bad file descriptor)
[2013-07-02 02:14:28.281302] E [changelog-helpers.c:442:changelog_fsync_thread] 0-master-changelog: failed to inject fsync event
[2013-07-02 02:14:33.286444] E [changelog-helpers.c:281:changelog_handle_change] 0-master-changelog: fsync failed (reason: Bad file descriptor)
[2013-07-02 02:14:33.286539] E [changelog-helpers.c:442:changelog_fsync_thread] 0-master-changelog: failed to inject fsync event
[2013-07-02 02:14:52.865433] E [changelog-helpers.c:161:changelog_rollover_changelog] 0-master-changelog: Failed to send file name to notify thread (reason: Resource temporarily unavailable)
[2013-07-02 02:14:52.865533] E [changelog-helpers.c:272:changelog_handle_change] 0-master-changelog: Problem rolling over changelog(s)
[2013-07-02 02:14:53.309218] E [changelog-helpers.c:281:changelog_handle_change] 0-master-changelog: fsync failed (reason: Bad file descriptor)
[2013-07-02 02:14:53.309354] E [changelog-helpers.c:442:changelog_fsync_thread] 0-master-changelog: failed to inject fsync event
Comment 7 Aravinda VK 2014-12-24 00:54:10 EST
Not seeing this behavior anymore. Hence closing this. Please re-open if observed again.

Note You need to log in before you can comment on or make changes to this bug.