Bug 1437244

Description of problem: Initial sync is successful, however, further filesystem changes are not detected or synced.

Version-Release number of selected component (if applicable): Both 3.8 & 3.10

How reproducible: 100%

Steps to Reproduce: Setup is comprised of two servers, both running fully updated CentOS 7. No SELinux.

* ill: Local server. Master volume named "foobar".
* aws: Remote server. Slave volume named "foobar".

* Both servers are running ntpd.
* Their clocks are in the same time zone and in sync.
* Passwordless SSH is setup and working.
* common_secret.pem.pub was generated on the local host "ill".
* 'create push-pem' was successfully on the local host "ill". Verified remote side had the two expected "command=" entries in ~/.ssh/authorized_keys.
* geo-rep 'start' session between two volumes was successfully created.
* The local filesystem successfully does an initial sync to the remote filesystem.
* No other changes are detected or synced

* 'status detail' looks clean but LAST_SYNCED never changes:

MASTER NODE: ill.franz.com
MASTER VOL: foobar
MASTER BRICK: /gv0/foobar
SLAVE USER: root
SLAVE NODE: aws.franz.com::foobar
STATUS: Active
CRAWL STATUS: Changelog Crawl
LAST_SYNCED: 2017-03-29 13:44:23


Actual results: No changes detected or synced


Expected results: Changes to be detected and synced


Additional info: I went with the default 'config' options. However, during troubleshooting I noticed that some of the default values seem suspicious/wrong. 

For example, 'remote_gsyncd' is set to '/nonexistent/gsyncd'.

Also, some of the variable values are incorrectly guessed. For example, 'gluster_log_file' is guessed as:

/var/log/glusterfs/geo-replication/foobar/ssh%3A%2F%2Froot%4054.165.144.9%3Agluster%3A%2F%2F127.0.0.1%3Afoobar.gluster.log

but the real file is:

/var/log/glusterfs/geo-replication/foobar/ssh%3A%2F%2Froot%4054.165.144.9%3Agluster%3A%2F%2F127.0.0.1%3Afoobar.%2Fgv0%2Ffoobar.gluster.log

I did try updating these variables to what seemed like more correct values but none of my changes had any effect on my problem.

I tried changing the 'change_detector' to xsync. No change in behavior.

I tried with both ext4 & xfs filesystems. No change in behavior.

I tried setting a checkpoint. No change in behavior.

I'm out of ideas at this point but happy to try anything & provide logs. Thanks so much for your time & help debugging!