Bug 1660225 - geo-rep does not replicate mv or rename of file
Summary: geo-rep does not replicate mv or rename of file
Keywords:
Status: CLOSED NEXTRELEASE
Alias: None
Product: GlusterFS
Classification: Community
Component: geo-replication
Version: 4.1
Hardware: aarch64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Sunny Kumar
QA Contact:
URL:
Whiteboard:
Depends On: 1583018
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-12-17 21:36 UTC by perplexed767
Modified: 2019-07-11 09:04 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-04-04 11:20:31 UTC
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:


Attachments (Terms of Use)
logs-master (352.32 KB, application/gzip)
2018-12-24 17:10 UTC, perplexed767
no flags Details
logs-slave (407.45 KB, application/gzip)
2018-12-24 17:11 UTC, perplexed767
no flags Details
additional info for test volume and geo session. (8.91 KB, text/plain)
2018-12-24 17:15 UTC, perplexed767
no flags Details


Links
System ID Priority Status Summary Last Updated
Gluster.org Gerrit 22476 None Merged cluster/dht: Fix rename journal in changelog 2019-04-04 11:20:30 UTC

Description perplexed767 2018-12-17 21:36:55 UTC
Description of problem: 

gluster 4.1.6 geo-replication is not replicating renames

Our application is basically a mail server.  We have mail stored on several (configurable) mount points, each having it's own volume (gluster)

The volumes are geo-replicated to a mirror server that is set up identically on the other site, the domain name basically being the only difference.

The mail application writes to a spool file, which is renamed when completed spooling.

the spool file still exists on the geo-slave and the spool file is still there, basically the rename does not get replicated.

the slave site the volumes are set to read-only, until we want to fail them over.

But to simplify the issue, I simply create a txt file on the root and immediately rename it, only the original name gets replicated, the rename is never done on the slave.

Version-Release number of selected component (if applicable):

4.1.6


How reproducible:

easy.


Steps to Reproduce:
1. create a file on the root mount of the volume
2. rename (mv) the file 
3. verify on slave that file has been renamed.

Actual results:

[vfeuk][xbesite1][fs-5][root@xbevmio01el-opco2-fs-25]: /glusterVol/.test/mfs_opco1_int_17 # echo "test 123" >>test4.txt;mv test4.txt test4_renamed.txt;ls -l test4.txt test4_renamed.txt
ls: cannot access 'test4.txt': No such file or directory
-rw-r----- 1 root root 9 Dec 17 22:15 test4_renamed.txt
[vfeuk][xbesite1][fs-5][root@xbevmio01el-opco2-fs-25]: /glusterVol/.test/mfs_opco1_int_17 # findmnt .
TARGET                             SOURCE                 FSTYPE         OPTIONS
/glusterVol/.test/mfs_opco1_int_17 fs-5:/mfs_opco1_int_17 fuse.glusterfs rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072


slave server:

[vfeuk][xmssite2][fs-5][root@xmsvmio02el-opco2-fs-25]: /glusterVol/.test/mfs_opco1_int_17 # ls -l test4.txt test4_renamed.txt
ls: cannot access 'test4_renamed.txt': No such file or directory
-rw-r----- 1 root root 9 Dec 17 22:15 test4.txt


Expected results:

file is renamed.

Additional info:

 gluster v geo mfs_opco1_int_17 fs-5-b.vfeuk.xmssite2.sero.gic.ericsson.se::mfs_opco1_int_17  status

MASTER NODE                                   MASTER VOL          MASTER BRICK                   SLAVE USER    SLAVE                                                           SLAVE NODE                                    STATUS     CRAWL STATUS       LAST_SYNCED
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
fs-5-b.vfeuk.xbesite1.sero.gic.ericsson.se    mfs_opco1_int_17    /exportg/mfs_opco1_int_17_b    root          fs-5-b.vfeuk.xmssite2.sero.gic.ericsson.se::mfs_opco1_int_17    fs-5-b.vfeuk.xmssite2.sero.gic.ericsson.se    Active     Changelog Crawl    2018-12-17 22:18:32
fs-6-b.vfeuk.xbesite1.sero.gic.ericsson.se    mfs_opco1_int_17    /exportg/mfs_opco1_int_17_b    root          fs-5-b.vfeuk.xmssite2.sero.gic.ericsson.se::mfs_opco1_int_17    fs-6-b.vfeuk.xmssite2.sero.gic.ericsson.se    Passive    N/A                N/A



 /exportg/mfs_opco1_int_17_b/internal # gluster v geo mfs_opco1_int_17 fs-5-b.vfeuk.xmssite2.sero.gic.ericsson.se::mfs_opco1_int_17 config
access_mount:false
allow_network:
change_detector:changelog
change_interval:5
changelog_archive_format:%Y%m
changelog_batch_size:727040
changelog_log_file:/var/log/glusterfs/geo-replication/mfs_opco1_int_17_fs-5-b.vfeuk.xmssite2.sero.gic.ericsson.se_mfs_opco1_int_17/changes-${local_id}.log
changelog_log_level:WARNING
checkpoint:1545081336
chnagelog_archive_format:%Y%m
cli_log_file:/var/log/glusterfs/geo-replication/cli.log
cli_log_level:INFO
connection_timeout:60
georep_session_working_dir:/var/lib/glusterd/geo-replication/mfs_opco1_int_17_fs-5-b.vfeuk.xmssite2.sero.gic.ericsson.se_mfs_opco1_int_17/
gluster_cli_options:
gluster_command:gluster
gluster_command_dir:/usr/sbin
gluster_log_file:/var/log/glusterfs/geo-replication/mfs_opco1_int_17_fs-5-b.vfeuk.xmssite2.sero.gic.ericsson.se_mfs_opco1_int_17/mnt-${local_id}.log
gluster_log_level:WARNING
gluster_logdir:/var/log/glusterfs
gluster_params:aux-gfid-mount acl
gluster_rundir:/var/run/gluster
glusterd_workdir:/var/lib/glusterd
gsyncd_miscdir:/var/lib/misc/gluster/gsyncd
ignore_deletes:false
isolated_slaves:
log_file:/var/log/glusterfs/geo-replication/mfs_opco1_int_17_fs-5-b.vfeuk.xmssite2.sero.gic.ericsson.se_mfs_opco1_int_17/gsyncd.log
log_level:WARNING
log_rsync_performance:true
master_disperse_count:1
master_replica_count:1
max_rsync_retries:10
meta_volume_mnt:/var/run/gluster/shared_storage
pid_file:/var/run/gluster/gsyncd-mfs_opco1_int_17-fs-5-b.vfeuk.xmssite2.sero.gic.ericsson.se-mfs_opco1_int_17.pid
remote_gsyncd:
replica_failover_interval:1
rsync_command:rsync
rsync_opt_existing:true
rsync_opt_ignore_missing_args:true
rsync_options:
rsync_ssh_options:
slave_access_mount:false
slave_gluster_command_dir:/usr/sbin
slave_gluster_log_file:/var/log/glusterfs/geo-replication-slaves/mfs_opco1_int_17_fs-5-b.vfeuk.xmssite2.sero.gic.ericsson.se_mfs_opco1_int_17/mnt-${master_node}-${master_brick_id}.log
slave_gluster_log_file_mbr:/var/log/glusterfs/geo-replication-slaves/mfs_opco1_int_17_fs-5-b.vfeuk.xmssite2.sero.gic.ericsson.se_mfs_opco1_int_17/mnt-mbr-${master_node}-${master_brick_id}.log
slave_gluster_log_level:INFO
slave_gluster_params:aux-gfid-mount acl
slave_log_file:/var/log/glusterfs/geo-replication-slaves/mfs_opco1_int_17_fs-5-b.vfeuk.xmssite2.sero.gic.ericsson.se_mfs_opco1_int_17/gsyncd.log
slave_log_level:INFO
slave_timeout:120
special_sync_mode:
ssh_command:ssh
ssh_options:-oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret.pem
ssh_options_tar:-oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/tar_ssh.pem
ssh_port:22
state_file:/var/lib/glusterd/geo-replication/mfs_opco1_int_17_fs-5-b.vfeuk.xmssite2.sero.gic.ericsson.se_mfs_opco1_int_17/monitor.status
state_socket_unencoded:
stime_xattr_prefix:trusted.glusterfs.9762dbff-67fc-41fa-b326-a327476869be.cf526ed9-a7a2-4f56-a94a-ba9e0d70d166
sync_acls:true
sync_jobs:1
sync_xattrs:true
tar_command:tar
use_meta_volume:true
use_rsync_xattrs:false
use_tarssh:false
working_dir:/var/lib/misc/gluster/gsyncd/mfs_opco1_int_17_fs-5-b.vfeuk.xmssite2.sero.gic.ericsson.se_mfs_opco1_int_17/


[vfeuk][xbesite1][fs-5][root@xbevmio01el-opco2-fs-25]: /exportg/mfs_opco1_int_17_b/internal # gluster v info mfs_opco1_int_17

Volume Name: mfs_opco1_int_17
Type: Replicate
Volume ID: 9762dbff-67fc-41fa-b326-a327476869be
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: fs-5-b.vfeuk.xbesite1.sero.gic.ericsson.se:/exportg/mfs_opco1_int_17_b
Brick2: fs-6-b.vfeuk.xbesite1.sero.gic.ericsson.se:/exportg/mfs_opco1_int_17_b
Options Reconfigured:
features.read-only: off
changelog.changelog: on
geo-replication.ignore-pid-check: on
geo-replication.indexing: on
diagnostics.client-log-level: WARNING
diagnostics.brick-log-level: WARNING
network.ping-timeout: 5
performance.write-behind-window-size: 64MB
server.allow-insecure: on
server.event-threads: 3
client.event-threads: 8
network.inode-lru-limit: 200000
performance.md-cache-timeout: 600
performance.cache-invalidation: on
performance.stat-prefetch: on
features.cache-invalidation-timeout: 600
features.cache-invalidation: on
performance.parallel-readdir: on
cluster.lookup-optimize: on
cluster.favorite-child-policy: mtime
performance.io-thread-count: 64
performance.readdir-ahead: on
performance.cache-size: 512MB
nfs.disable: on
cluster.enable-shared-storage: enable
[vfeuk][xbesite1][fs-5][root@xbevmio01el-opco2-fs-25]: /exportg/mfs_opco1_int_17_b/internal # gluster v status mfs_opco1_int_17
Status of volume: mfs_opco1_int_17
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick fs-5-b.vfeuk.xbesite1.sero.gic.ericss
on.se:/exportg/mfs_opco1_int_17_b           49159     0          Y       28908
Brick fs-6-b.vfeuk.xbesite1.sero.gic.ericss
on.se:/exportg/mfs_opco1_int_17_b           49159     0          Y       3611
Self-heal Daemon on localhost               N/A       N/A        Y       22663
Self-heal Daemon on fs-6-b.vfeuk.xbesite1.s
ero.gic.ericsson.se                         N/A       N/A        Y       14227

Task Status of Volume mfs_opco1_int_17
------------------------------------------------------------------------------
There are no active volume task


[vfeuk][xbesite1][fs-5][root@xbevmio01el-opco2-fs-25]: /exportg/mfs_opco1_int_17_b/internal # gluster peer status
Number of Peers: 1

Hostname: fs-6-b.vfeuk.xbesite1.sero.gic.ericsson.se
Uuid: 5c8b7596-840a-48e9-a3fe-e9ced3a48df0
State: Peer in Cluster (Connected)


gluster volume statedump mfs_opco1_int_17
Segmentation fault (core dumped)


 xfs_info /exportg/mfs_opco1_int_17_b
meta-data=/dev/mapper/glustervg-lvm_mfs_opco1_int_17_b isize=1024   agcount=16, agsize=163808 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=0        finobt=0 spinodes=0
data     =                       bsize=4096   blocks=2620928, imaxpct=33
         =                       sunit=32     swidth=32 blks
naming   =version 2              bsize=8192   ascii-ci=0 ftype=1
log      =internal               bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=32 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0


getfattr -d -m. -ehex /exportg/mfs_opco1_int_17_b
getfattr: Removing leading '/' from absolute path names
# file: exportg/mfs_opco1_int_17_b
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.9762dbff-67fc-41fa-b326-a327476869be.cf526ed9-a7a2-4f56-a94a-ba9e0d70d166.entry_stime=0x5c18140100000000
trusted.glusterfs.9762dbff-67fc-41fa-b326-a327476869be.cf526ed9-a7a2-4f56-a94a-ba9e0d70d166.stime=0x5c18140100000000
trusted.glusterfs.9762dbff-67fc-41fa-b326-a327476869be.xtime=0x5c18140e000c9211
trusted.glusterfs.dht=0x000000010000000000000000ffffffff
trusted.glusterfs.volume-id=0x9762dbff67fc41fab326a327476869be



[vfeuk][xbesite1][fs-5][root@xbevmio01el-opco2-fs-25]: /exportg/mfs_opco1_int_17_b/internal # uname -r; cat /etc/issue
4.4.103-6.38-default

Welcome to SUSE Linux Enterprise Server 12 SP3  (x86_64) - Kernel \r (\l).



[vfeuk][xmssite2][fs-5][root@xmsvmio02el-opco2-fs-25]: /glusterVol/.test/mfs_opco1_int_17 # df -Th
Filesystem                                             Type            Size  Used Avail Use% Mounted on
devtmpfs                                               devtmpfs        9.8G   12K  9.8G   1% /dev
tmpfs                                                  tmpfs           9.9G  4.0K  9.9G   1% /dev/shm
tmpfs                                                  tmpfs           9.9G  483M  9.4G   5% /run
tmpfs                                                  tmpfs           9.9G     0  9.9G   0% /sys/fs/cgroup
/dev/sda2                                              btrfs            32G  1.6G   29G   6% /
/dev/sda1                                              ext3            259M   46M  200M  19% /boot
tmpfs                                                  tmpfs           2.0G     0  2.0G   0% /run/user/0
/dev/mapper/glustervg-lvm_glusterVarLog                xfs              50G  232M   50G   1% /exportg/gluster_var_log
/dev/mapper/glustervg-lvm_mfs_opco1_int_10_b           xfs              10G  4.7G  5.4G  47% /exportg/mfs_opco1_int_10_b
/dev/mapper/glustervg-lvm_mfs_opco1_int_11_b           xfs              10G  446M  9.6G   5% /exportg/mfs_opco1_int_11_b
/dev/mapper/glustervg-lvm_mfs_opco1_int_12_b           xfs              10G  4.6G  5.5G  46% /exportg/mfs_opco1_int_12_b
/dev/mapper/glustervg-lvm_mfs_opco1_int_13_b           xfs              10G  1.1G  9.0G  11% /exportg/mfs_opco1_int_13_b
/dev/mapper/glustervg-lvm_mfs_opco1_int_14_b           xfs              10G  518M  9.5G   6% /exportg/mfs_opco1_int_14_b
/dev/mapper/glustervg-lvm_mfs_opco1_int_15_b           xfs              10G  1.2G  8.8G  12% /exportg/mfs_opco1_int_15_b
/dev/mapper/glustervg-lvm_mfs_opco1_int_16_b           xfs              10G  637M  9.4G   7% /exportg/mfs_opco1_int_16_b
/dev/mapper/glustervg-lvm_mfs_opco1_int_17_b           xfs              10G  2.6G  7.5G  26% /exportg/mfs_opco1_int_17_b
/dev/mapper/glustervg-lvm_mfs_opco1_int_18_b           xfs              10G   38M   10G   1% /exportg/mfs_opco1_int_18_b
/dev/mapper/glustervg-lvm_mfs_opco1_int_19_b           xfs              10G   38M   10G   1% /exportg/mfs_opco1_int_19_b
/dev/mapper/glustervg-lvm_mfs_opco1_int_1a_b           xfs              10G   38M   10G   1% /exportg/mfs_opco1_int_1a_b
/dev/mapper/glustervg-lvm_mfs_opco1_int_1b_b           xfs              10G   38M   10G   1% /exportg/mfs_opco1_int_1b_b
/dev/mapper/glustervg-lvm_mfs_opco1_int_1c_b           xfs              10G   38M   10G   1% /exportg/mfs_opco1_int_1c_b
/dev/mapper/glustervg-lvm_mfs_opco1_int_1d_b           xfs              10G   42M   10G   1% /exportg/mfs_opco1_int_1d_b
/dev/mapper/glustervg-lvm_mfs_opco1_int_1e_b           xfs              10G   42M   10G   1% /exportg/mfs_opco1_int_1e_b
/dev/mapper/glustervg-lvm_mfs_opco1_int_1f_b           xfs              10G   38M   10G   1% /exportg/mfs_opco1_int_1f_b
10.221.81.224:/gluster_shared_storage                  fuse.glusterfs   32G  2.1G   29G   7% /run/gluster/shared_storage
om-1-b.vfeuk.xmssite2.sero.gic.ericsson.se:/gcluster   fuse.glusterfs   25G  957M   25G   4% /cluster
fs-5:/gluster_shared_storage                           fuse.glusterfs   32G  2.1G   29G   7% /glusterVol/.test/gluster_shared_storage
fs-5:/mfs_opco1_int_10                                 fuse.glusterfs   10G  4.8G  5.3G  48% /glusterVol/.test/mfs_opco1_int_10
fs-5:/mfs_opco1_int_11                                 fuse.glusterfs   10G  548M  9.5G   6% /glusterVol/.test/mfs_opco1_int_11
fs-5:/mfs_opco1_int_12                                 fuse.glusterfs   10G  4.7G  5.4G  47% /glusterVol/.test/mfs_opco1_int_12
fs-5:/mfs_opco1_int_13                                 fuse.glusterfs   10G  1.2G  8.9G  12% /glusterVol/.test/mfs_opco1_int_13
fs-5:/mfs_opco1_int_14                                 fuse.glusterfs   10G  620M  9.4G   7% /glusterVol/.test/mfs_opco1_int_14
fs-5:/mfs_opco1_int_15                                 fuse.glusterfs   10G  1.3G  8.7G  13% /glusterVol/.test/mfs_opco1_int_15
fs-5:/mfs_opco1_int_16                                 fuse.glusterfs   10G  740M  9.3G   8% /glusterVol/.test/mfs_opco1_int_16
fs-5:/mfs_opco1_int_17                                 fuse.glusterfs   10G  2.7G  7.4G  27% /glusterVol/.test/mfs_opco1_int_17
fs-5:/mfs_opco1_int_18                                 fuse.glusterfs   10G  140M  9.9G   2% /glusterVol/.test/mfs_opco1_int_18
fs-5:/mfs_opco1_int_19                                 fuse.glusterfs   10G  144M  9.9G   2% /glusterVol/.test/mfs_opco1_int_19
fs-5:/mfs_opco1_int_1a                                 fuse.glusterfs   10G  144M  9.9G   2% /glusterVol/.test/mfs_opco1_int_1a
fs-5:/mfs_opco1_int_1b                                 fuse.glusterfs   10G  144M  9.9G   2% /glusterVol/.test/mfs_opco1_int_1b
fs-5:/mfs_opco1_int_1c                                 fuse.glusterfs   10G  144M  9.9G   2% /glusterVol/.test/mfs_opco1_int_1c
fs-5:/mfs_opco1_int_1d                                 fuse.glusterfs   10G  144M  9.9G   2% /glusterVol/.test/mfs_opco1_int_1d
fs-5:/mfs_opco1_int_1e                                 fuse.glusterfs   10G  144M  9.9G   2% /glusterVol/.test/mfs_opco1_int_1e
fs-5:/mfs_opco1_int_1f                                 fuse.glusterfs   10G  144M  9.9G   2% /glusterVol/.test/mfs_opco1_int_1f
fs-1-b.vfeuk.xmssite2.sero.gic.ericsson.se:/perf_opco1 fuse.glusterfs   16G  625M   16G   4% /glusterVol/perf_opco1
fs-1-b.vfeuk.xmssite2.sero.gic.ericsson.se:/cps_opco1  fuse.glusterfs  2.0G   64M  2.0G   4% /opt/global/cps
fs-1-b.vfeuk.xmssite2.sero.gic.ericsson.se:/logs_opco1 fuse.glusterfs   50G  2.1G   48G   5% /glusterVol/logs_opco1


==================

If any logs required please ask.

Comment 1 Sunny Kumar 2018-12-19 19:25:50 UTC
Hi,

Please share- 1. gluster log 2. geo-replication log from master and 3. mount log from slave.

- Sunny

Comment 2 perplexed767 2018-12-24 16:03:38 UTC
(In reply to Sunny Kumar from comment #1)
> Hi,
> 
> Please share- 1. gluster log 2. geo-replication log from master and 3. mount
> log from slave.
> 
> - Sunny

How do I attach logs, or do you wish I email them to you?

Comment 3 perplexed767 2018-12-24 17:10:36 UTC
Created attachment 1516584 [details]
logs-master

logs from master, geo master, brick glustredd, mount logs and changelogs from brick.

Comment 4 perplexed767 2018-12-24 17:11:44 UTC
Created attachment 1516585 [details]
logs-slave

geo slave logs, brick logs, mount logs, glusterd logs

Comment 5 perplexed767 2018-12-24 17:15:27 UTC
Created attachment 1516586 [details]
additional info for test volume and geo session.

new test volume and replication session to create requested logs.

Same server, same brick type, lvm xfs, save servers as in description.

Comment 6 perplexed767 2018-12-24 17:21:33 UTC
Never mind found how to attach logs.

I created a new volume and replication session to reproduce the logs you requested.

As the other sessions are not created at the moment as they are turned off to do other tests.

However same result and  very similar setup, except debug logs are turned on and smaller bricks.

Comment 7 perplexed767 2019-02-06 22:13:59 UTC
(In reply to Sunny Kumar from comment #1)
> Hi,
> 
> Please share- 1. gluster log 2. geo-replication log from master and 3. mount
> log from slave.
> 
> - Sunny

Is there any news on this bug? It has been over a month since I filed it.

Comment 8 Sunny Kumar 2019-02-25 10:25:13 UTC
Hi,

Looks like https://review.gluster.org/#/c/glusterfs/+/20093/.

But I am trying for reproducer to analyse more.

If something is missing in step to reproduce please add.

- Sunny

Comment 9 perplexed767 2019-02-26 15:22:23 UTC
(In reply to Sunny Kumar from comment #8)
> Hi,
> 
> Looks like https://review.gluster.org/#/c/glusterfs/+/20093/.
> 
> But I am trying for reproducer to analyse more.
> 
> If something is missing in step to reproduce please add.
> 
> - Sunny

sounds like it could be, which version of gluster is this released in?

Comment 10 perplexed767 2019-02-26 15:35:18 UTC
It is quite easy to reproduce, so don't think I missed anything.

default config, replica 2 on both sites, create geo session, start, create and rename a file.

Comment 11 asender@testlabs.com.au 2019-04-02 04:37:39 UTC
You can try this simple test to reproduce the problem.

On Master
[svc_sp_st_script@hplispnfs30079 conf]$ touch test.txt
[svc_sp_st_script@hplispnfs30079 conf]$ vi test.txt 
a
b
c
d

[svc_sp_st_script@hplispnfs30079 conf]$ ll test.txt 
-rw-r----- 1 svc_sp_st_script domain users 8 Apr  2 14:59 test.txt


On Slave
[root@hplispnfs40079 conf]# ll test.txt 
-rw-r----- 1 svc_sp_st_script domain users 8 Apr  2 14:59 test.txt

[root@hplispnfs40079 conf]# cat test.txt 
a
b
c
d

On Master
[svc_sp_st_script@hplispnfs30079 conf]$ mv test.txt test-moved.txt
[svc_sp_st_script@hplispnfs30079 conf]$ ll test-moved.txt 
-rw-r----- 1 svc_sp_st_script domain users 8 Apr  2 14:59 test-moved.txt

On Slave
File is not deleted, test-moved.txt does not exist and is not replicated.

[root@hplispnfs40079 conf]# ll testfile 
-rw-r----- 1 svc_sp_st_script domain users 6 Apr  2 14:52 testfile

Comment 12 asender@testlabs.com.au 2019-04-02 04:38:35 UTC
I also tried setting use_tarssh:true but this did not change the behavior. 

[root@hplispnfs30079 conf]# gluster volume geo-replication common hplispnfs40079::common config
access_mount:false
allow_network:
change_detector:changelog
change_interval:5
changelog_archive_format:%Y%m
changelog_batch_size:727040
changelog_log_file:/var/log/glusterfs/geo-replication/common_hplispnfs40079_common/changes-${local_id}.log
changelog_log_level:INFO
checkpoint:0
chnagelog_archive_format:%Y%m
cli_log_file:/var/log/glusterfs/geo-replication/cli.log
cli_log_level:INFO
connection_timeout:60
georep_session_working_dir:/var/lib/glusterd/geo-replication/common_hplispnfs40079_common/
gluster_cli_options:
gluster_command:gluster
gluster_command_dir:/usr/sbin
gluster_log_file:/var/log/glusterfs/geo-replication/common_hplispnfs40079_common/mnt-${local_id}.log
gluster_log_level:INFO
gluster_logdir:/var/log/glusterfs
gluster_params:aux-gfid-mount acl
gluster_rundir:/var/run/gluster
glusterd_workdir:/var/lib/glusterd
gsyncd_miscdir:/var/lib/misc/gluster/gsyncd
ignore_deletes:false
isolated_slaves:
log_file:/var/log/glusterfs/geo-replication/common_hplispnfs40079_common/gsyncd.log
log_level:INFO
log_rsync_performance:false
master_disperse_count:1
master_replica_count:1
max_rsync_retries:10
meta_volume_mnt:/var/run/gluster/shared_storage
pid_file:/var/run/gluster/gsyncd-common-hplispnfs40079-common.pid
remote_gsyncd:/usr/libexec/glusterfs/gsyncd
replica_failover_interval:1
rsync_command:rsync
rsync_opt_existing:true
rsync_opt_ignore_missing_args:true
rsync_options:
rsync_ssh_options:
slave_access_mount:false
slave_gluster_command_dir:/usr/sbin
slave_gluster_log_file:/var/log/glusterfs/geo-replication-slaves/common_hplispnfs40079_common/mnt-${master_node}-${master_brick_id}.log
slave_gluster_log_file_mbr:/var/log/glusterfs/geo-replication-slaves/common_hplispnfs40079_common/mnt-mbr-${master_node}-${master_brick_id}.log
slave_gluster_log_level:INFO
slave_gluster_params:aux-gfid-mount acl
slave_log_file:/var/log/glusterfs/geo-replication-slaves/common_hplispnfs40079_common/gsyncd.log
slave_log_level:INFO
slave_timeout:120
special_sync_mode:
ssh_command:ssh
ssh_options:-oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret.pem
ssh_options_tar:-oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/tar_ssh.pem
ssh_port:22
state_file:/var/lib/glusterd/geo-replication/common_hplispnfs40079_common/monitor.status
state_socket_unencoded:
stime_xattr_prefix:trusted.glusterfs.bb691a2e-801c-435b-a905-11ad249d43a7.ab3b208f-8cd1-4a2d-bf56-4a98434605c5
sync_acls:true
sync_jobs:3
sync_xattrs:true
tar_command:tar
use_meta_volume:true
use_rsync_xattrs:false
use_tarssh:true
working_dir:/var/lib/misc/gluster/gsyncd/common_hplispnfs40079_common/

Comment 13 Kotresh HR 2019-04-02 06:48:46 UTC
This issue is fixed in upstream and 5.x and 6.x series

Patch: https://review.gluster.org/#/c/glusterfs/+/20093/

Comment 14 Kotresh HR 2019-04-02 06:49:42 UTC
Workaround:
The issue affects only single distribute volumes i.e 1*2 and 1*3 volumes.
It doesn't affect n*2 or n*3 volumes where n>1. So one way to fix is to convert
single distribute to two distribute volume  or upgrade to later versions
if it can't be waited until next 4.1.x release.

Comment 15 Worker Ant 2019-04-02 06:55:57 UTC
REVIEW: https://review.gluster.org/22476 (cluster/dht: Fix rename journal in changelog) posted (#1) for review on release-4.1 by Kotresh HR

Comment 16 asender@testlabs.com.au 2019-04-03 04:16:17 UTC
(In reply to Kotresh HR from comment #13)
> This issue is fixed in upstream and 5.x and 6.x series
> 
> Patch: https://review.gluster.org/#/c/glusterfs/+/20093/

We are having the issue in replicate mode (using replica 2). 

Adrian Sender

Comment 17 perplexed767 2019-04-03 21:55:30 UTC
(In reply to Kotresh HR from comment #14)
> Workaround:
> The issue affects only single distribute volumes i.e 1*2 and 1*3 volumes.
> It doesn't affect n*2 or n*3 volumes where n>1. So one way to fix is to
> convert
> single distribute to two distribute volume  or upgrade to later versions
> if it can't be waited until next 4.1.x release.

greate thanks, is it planned to be backported to for 4.x as my os (sles 12.2) does not currenty support 5.x gluster) I would have to upgrade the os to sles 12.3

Comment 18 Kotresh HR 2019-04-04 06:10:30 UTC
I have backported the patch https://review.gluster.org/#/c/glusterfs/+/22476/. It's not merged yet.

Comment 19 Worker Ant 2019-04-04 11:20:31 UTC
REVIEW: https://review.gluster.org/22476 (cluster/dht: Fix rename journal in changelog) merged (#1) on release-4.1 by Kotresh HR

Comment 20 turnerb 2019-05-15 22:58:58 UTC
I was curious if anyone ever got this resolved? I was running 4.1.7 and set up a geo-replica, this had the above issue with the renaming of files and directories. I have tired up grading to 4.1.8 and have now moved to 5.6 and best I have now is replicated renames of directories. Renaming of files still doesn't get replicated to the geo-replica volume.

Comment 21 asender@testlabs.com.au 2019-05-15 23:09:33 UTC
Some custom compiled RPMS are versioned 4.1.8-0.1.git... and contain the fixes. What a mess this project has become.  Broken all versions.

Official it will be in 4.1.9 but you can use the below RPMS - we are running it again in production now and appears to be ok. Make sure you update clients as well.


[1] RPMs for 4.1 including the fix for el7:
https://build.gluster.org/job/rpm-el7/3599/artifact/

-Adrian Sender

Comment 22 turnerb 2019-05-16 08:27:43 UTC
Thank you Adrian, appreciate the feedback. Unfortunatley that URL returns me a 404 error so cannot get that installed. 

I may just wait for the 4.1.9 release to go GA, seems based on the release cycle that it mught well be out next week. Unless you happen to have a copy of the RPM's that you can share? 

Thanks, Ben.

Comment 23 hari gowtham 2019-07-11 09:04:50 UTC
This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-4.1.9, please open a new bug report.

glusterfs-4.1.9 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution.

[1] https://lists.gluster.org/pipermail/gluster-users/2019-June/036679.html
[2] https://www.gluster.org/pipermail/gluster-users/


Note You need to log in before you can comment on or make changes to this bug.