Bug 1483541

Summary: [geo-rep]: Slave has more entries than Master in multiple hardlink/rename scenario
Product: [Red Hat Storage] Red Hat Gluster Storage Reporter: Rahul Hinduja <rhinduja>
Component: geo-replicationAssignee: Kotresh HR <khiremat>
Status: CLOSED ERRATA QA Contact: Rochelle <rallan>
Severity: high Docs Contact:
Priority: high    
Version: rhgs-3.3CC: amukherj, apaladug, atumball, csaba, khiremat, rallan, rcyriac, rhinduja, rhs-bugs, sheggodu, storage-qa-internal
Target Milestone: ---Flags: khiremat: needinfo-
Target Release: RHGS 3.4.0   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-09-04 06:35:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1599587    
Bug Blocks: 1503134    

Description Rahul Hinduja 2017-08-21 11:24:21 UTC
Description of problem:
=======================

During a scenario where lot of hardlink and renames of same files were tried while one of the brick is down. The end results shows additional files at slave:

Following are two addtional files at slave on a single directory: 

At Slave:
---------

[root@dj ~]# ls /mnt/slave/f/kill-scenario/thread0/level01/level11/level21/level31/level41/level51/level61/level71/level81/level91/ | grep 5998f908%%C7SK0C8YNY
5998f908%%C7SK0C8YNY
[root@dj ~]# ls /mnt/slave/f/kill-scenario/thread0/level01/level11/level21/level31/level41/level51/level61/level71/level81/level91/ | grep 5998f908%%DMB81F7NUA
5998f908%%DMB81F7NUA
[root@dj ~]# 

At Master:
----------

[root@dj ~]# ls /mnt/master/f/kill-scenario/thread0/level01/level11/level21/level31/level41/level51/level61/level71/level81/level91/ | grep 5998f908%%C7SK0C8YNY
[root@dj ~]# ls /mnt/master/f/kill-scenario/thread0/level01/level11/level21/level31/level41/level51/level61/level71/level81/level91/ | grep 5998f908%%DMB81F7NUA
[root@dj ~]# 

Geo-Replication logs shows the Entry Failure for these files during rename as:

[2017-08-20 16:28:52.920265] I [master(/rhs/brick1/b1):1203:crawl] _GMaster: slave's time: (1503243034, 0)
[2017-08-20 16:28:52.987740] I [master(/rhs/brick2/b5):1203:crawl] _GMaster: slave's time: (1503243095, 0)
[2017-08-20 16:29:16.379971] I [master(/rhs/brick3/b9):1203:crawl] _GMaster: slave's time: (1503243396, 0)
[2017-08-20 16:31:56.771578] E [master(/rhs/brick3/b9):782:log_failures] _GMaster: ENTRY FAILED: ({'stat': {'atime': 1503243030.460567, 'gid': 0, 'mtime': 1503243210.3251548, 'mode': 32769, 'uid': 0}, 'entry1': '.gfid/16eb8f74-2295-4f18-8e82-64ed85999c81/5998f947%%Y2TXWBTNNE', 'gfid': '1bdb8a40-0abb-4471-9348-dae93fcb4528', 'link': None, 'entry': '.gfid/16eb8f74-2295-4f18-8e82-64ed85999c81/5998f908%%C7SK0C8YNY', 'op': 'RENAME'}, 2)
[2017-08-20 16:31:56.774578] E [master(/rhs/brick3/b9):782:log_failures] _GMaster: ENTRY FAILED: ({'stat': {'atime': 1503243030.4875665, 'gid': 0, 'mtime': 1503243210.3361545, 'mode': 33161, 'uid': 0}, 'entry1': '.gfid/16eb8f74-2295-4f18-8e82-64ed85999c81/5998f947%%M3HN9AVSW9', 'gfid': 'b55ee2c7-0831-4f54-a462-cdbf77377aa6', 'link': None, 'entry': '.gfid/16eb8f74-2295-4f18-8e82-64ed85999c81/5998f908%%DMB81F7NUA', 'op': 'RENAME'}, 2)
[2017-08-20 16:31:56.777143] E [master(/rhs/brick3/b9):782:log_failures] _GMaster: ENTRY FAILED: ({'stat': {'atime': 1503243030.6145642, 'gid': 0, 'mtime': 1503243210.461152, 'mode': 33242, 'uid': 0}, 'entry1': '.gfid/f2f823c5-a8df-44ff-839a-5aa6056fd47d/5998f948%%RSV6VIJG56', 'gfid': '7ac38ace-02ef-4312-9cd3-8f431df2d258', 'link': None, 'entry': '.gfid/f2f823c5-a8df-44ff-839a-5aa6056fd47d/5998f908%%15AWA4KG78', 'op': 'RENAME'}, 2)


Initial State:
==============

[root@dhcp37-64 master]# gluster volume geo-replication master 10.70.43.82::slave status
 
MASTER NODE     MASTER VOL    MASTER BRICK       SLAVE USER    SLAVE                 SLAVE NODE      STATUS     CRAWL STATUS       LAST_SYNCED                  
-----------------------------------------------------------------------------------------------------------------------------------------------------
10.70.37.64     master        /rhs/brick1/b1     root          10.70.43.82::slave    10.70.42.39     Active     Changelog Crawl    2017-08-20 21:00:19          
10.70.37.64     master        /rhs/brick2/b5     root          10.70.43.82::slave    10.70.42.39     Active     Changelog Crawl    2017-08-20 21:00:20          
10.70.37.64     master        /rhs/brick3/b9     root          10.70.43.82::slave    10.70.42.39     Active     Changelog Crawl    2017-08-20 21:00:21          
10.70.37.133    master        /rhs/brick1/b3     root          10.70.43.82::slave    10.70.43.82     Active     Changelog Crawl    2017-08-20 21:00:23          
10.70.37.133    master        /rhs/brick2/b7     root          10.70.43.82::slave    10.70.43.82     Active     Changelog Crawl    2017-08-20 21:00:24          
10.70.37.129    master        /rhs/brick1/b4     root          10.70.43.82::slave    10.70.43.101    Passive    N/A                N/A                          
10.70.37.129    master        /rhs/brick2/b8     root          10.70.43.82::slave    10.70.43.101    Passive    N/A                N/A                          
10.70.37.60     master        /rhs/brick1/b2     root          10.70.43.82::slave    10.70.43.190    Passive    N/A                N/A                          
10.70.37.60     master        /rhs/brick2/b6     root          10.70.43.82::slave    10.70.43.190    Passive    N/A                N/A                          
10.70.37.60     master        /rhs/brick3/b10    root          10.70.43.82::slave    10.70.43.190    Passive    N/A                N/A                          
[root@dhcp37-64 master]# 

Scenario:
+++++++++

Start writing data at mount:
============================

for i in {create,chmod,hardlink,chgrp,symlink,hardlink,truncate,hardlink,rename,hardlink,symlink,hardlink,rename,chown,rename,create,hardlink,hardlink,symlink,rename}; do crefi --multi -n 5 -b 10 -d 10 --max=10K --min=500 --random -T 10 -t text --fop=$i /mnt/master/f/kill-scenario/ ; sleep 10 ; done

Kill brick b1 => Wait for status to faulty and b2 to become Active (History Crawl)
Kill brick b5 => Wait for status to faulty and b2 to become Active (History Crawl)
Kill brick b9 => Wait for status to faulty and b10 to become Active (History Crawl)

Data completed in between..
===========================

Stop the geo-replication...
===========================

Bring back the offline bricks using start force:
================================================

Wait for heal to complete:
==========================

[root@dhcp37-64 master]# gluster volume heal master info 
Brick 10.70.37.64:/rhs/brick1/b1
Status: Connected
Number of entries: 0

Brick 10.70.37.60:/rhs/brick1/b2
Status: Connected
Number of entries: 0

Brick 10.70.37.133:/rhs/brick1/b3
Status: Connected
Number of entries: 0

Brick 10.70.37.129:/rhs/brick1/b4
Status: Connected
Number of entries: 0

Brick 10.70.37.64:/rhs/brick2/b5
Status: Connected
Number of entries: 0

Brick 10.70.37.60:/rhs/brick2/b6
Status: Connected
Number of entries: 0

Brick 10.70.37.133:/rhs/brick2/b7
Status: Connected
Number of entries: 0

Brick 10.70.37.129:/rhs/brick2/b8
Status: Connected
Number of entries: 0

Brick 10.70.37.64:/rhs/brick3/b9
Status: Connected
Number of entries: 0

Brick 10.70.37.60:/rhs/brick3/b10
Status: Connected
Number of entries: 0

[root@dhcp37-64 master]# 
[root@dhcp37-64 master]# 
[root@dhcp37-64 master]# gluster volume status master
Status of volume: master
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.37.64:/rhs/brick1/b1            49156     0          Y       14161
Brick 10.70.37.60:/rhs/brick1/b2            49152     0          Y       13358
Brick 10.70.37.133:/rhs/brick1/b3           49152     0          Y       13371
Brick 10.70.37.129:/rhs/brick1/b4           49152     0          Y       13364
Brick 10.70.37.64:/rhs/brick2/b5            49157     0          Y       14182
Brick 10.70.37.60:/rhs/brick2/b6            49153     0          Y       13377
Brick 10.70.37.133:/rhs/brick2/b7           49153     0          Y       13390
Brick 10.70.37.129:/rhs/brick2/b8           49153     0          Y       13383
Brick 10.70.37.64:/rhs/brick3/b9            49158     0          Y       14203
Brick 10.70.37.60:/rhs/brick3/b10           49154     0          Y       13396
NFS Server on localhost                     2049      0          Y       14226
Self-heal Daemon on localhost               N/A       N/A        Y       14236
NFS Server on 10.70.37.133                  2049      0          Y       4019 
Self-heal Daemon on 10.70.37.133            N/A       N/A        Y       4028 
NFS Server on 10.70.37.129                  2049      0          Y       32227
Self-heal Daemon on 10.70.37.129            N/A       N/A        Y       32236
NFS Server on 10.70.37.60                   2049      0          Y       1916 
Self-heal Daemon on 10.70.37.60             N/A       N/A        Y       1925 
 
Task Status of Volume master
------------------------------------------------------------------------------
There are no active volume tasks
 
[root@dhcp37-64 master]# 
[root@dhcp37-64 master]# 
[root@dhcp37-64 master]# gluster volume geo-replication master 10.70.43.82::slave status
 
MASTER NODE     MASTER VOL    MASTER BRICK       SLAVE USER    SLAVE                 SLAVE NODE    STATUS     CRAWL STATUS    LAST_SYNCED          
----------------------------------------------------------------------------------------------------------------------------------------
10.70.37.64     master        /rhs/brick1/b1     root          10.70.43.82::slave    N/A           Stopped    N/A             N/A                  
10.70.37.64     master        /rhs/brick2/b5     root          10.70.43.82::slave    N/A           Stopped    N/A             N/A                  
10.70.37.64     master        /rhs/brick3/b9     root          10.70.43.82::slave    N/A           Stopped    N/A             N/A                  
10.70.37.133    master        /rhs/brick1/b3     root          10.70.43.82::slave    N/A           Stopped    N/A             N/A                  
10.70.37.133    master        /rhs/brick2/b7     root          10.70.43.82::slave    N/A           Stopped    N/A             N/A                  
10.70.37.129    master        /rhs/brick1/b4     root          10.70.43.82::slave    N/A           Stopped    N/A             N/A                  
10.70.37.129    master        /rhs/brick2/b8     root          10.70.43.82::slave    N/A           Stopped    N/A             N/A                  
10.70.37.60     master        /rhs/brick1/b2     root          10.70.43.82::slave    N/A           Stopped    N/A             N/A                  
10.70.37.60     master        /rhs/brick2/b6     root          10.70.43.82::slave    N/A           Stopped    N/A             N/A                  
10.70.37.60     master        /rhs/brick3/b10    root          10.70.43.82::slave    N/A           Stopped    N/A             N/A                  


Start the geo-replication:
==========================

[root@dhcp37-64 master]# gluster volume geo-replication master 10.70.43.82::slave start
Starting geo-replication session between master & 10.70.43.82::slave has been successful
[root@dhcp37-64 master]# gluster volume geo-replication master 10.70.43.82::slave status
 
MASTER NODE     MASTER VOL    MASTER BRICK       SLAVE USER    SLAVE                 SLAVE NODE    STATUS             CRAWL STATUS    LAST_SYNCED          
------------------------------------------------------------------------------------------------------------------------------------------------
10.70.37.64     master        /rhs/brick1/b1     root          10.70.43.82::slave    N/A           Initializing...    N/A             N/A                  
10.70.37.64     master        /rhs/brick2/b5     root          10.70.43.82::slave    N/A           Initializing...    N/A             N/A                  
10.70.37.64     master        /rhs/brick3/b9     root          10.70.43.82::slave    N/A           Initializing...    N/A             N/A                  
10.70.37.129    master        /rhs/brick1/b4     root          10.70.43.82::slave    N/A           Initializing...    N/A             N/A                  
10.70.37.129    master        /rhs/brick2/b8     root          10.70.43.82::slave    N/A           Initializing...    N/A             N/A                  
10.70.37.133    master        /rhs/brick1/b3     root          10.70.43.82::slave    N/A           Initializing...    N/A             N/A                  
10.70.37.133    master        /rhs/brick2/b7     root          10.70.43.82::slave    N/A           Initializing...    N/A             N/A                  
10.70.37.60     master        /rhs/brick1/b2     root          10.70.43.82::slave    N/A           Initializing...    N/A             N/A                  
10.70.37.60     master        /rhs/brick2/b6     root          10.70.43.82::slave    N/A           Initializing...    N/A             N/A                  
10.70.37.60     master        /rhs/brick3/b10    root          10.70.43.82::slave    N/A           Initializing...    N/A             N/A                  
[root@dhcp37-64 master]# 
[root@dhcp37-64 master]# 
[root@dhcp37-64 master]# gluster volume geo-replication master 10.70.43.82::slave status
 
MASTER NODE     MASTER VOL    MASTER BRICK       SLAVE USER    SLAVE                 SLAVE NODE    STATUS             CRAWL STATUS    LAST_SYNCED          
------------------------------------------------------------------------------------------------------------------------------------------------
10.70.37.64     master        /rhs/brick1/b1     root          10.70.43.82::slave    N/A           Initializing...    N/A             N/A                  
10.70.37.64     master        /rhs/brick2/b5     root          10.70.43.82::slave    N/A           Initializing...    N/A             N/A                  
10.70.37.64     master        /rhs/brick3/b9     root          10.70.43.82::slave    N/A           Initializing...    N/A             N/A                  
10.70.37.133    master        /rhs/brick1/b3     root          10.70.43.82::slave    N/A           Initializing...    N/A             N/A                  
10.70.37.133    master        /rhs/brick2/b7     root          10.70.43.82::slave    N/A           Initializing...    N/A             N/A                  
10.70.37.129    master        /rhs/brick1/b4     root          10.70.43.82::slave    N/A           Initializing...    N/A             N/A                  
10.70.37.129    master        /rhs/brick2/b8     root          10.70.43.82::slave    N/A           Initializing...    N/A             N/A                  
10.70.37.60     master        /rhs/brick1/b2     root          10.70.43.82::slave    N/A           Initializing...    N/A             N/A                  
10.70.37.60     master        /rhs/brick2/b6     root          10.70.43.82::slave    N/A           Initializing...    N/A             N/A                  
10.70.37.60     master        /rhs/brick3/b10    root          10.70.43.82::slave    N/A           Initializing...    N/A             N/A                  
[root@dhcp37-64 master]# 
[root@dhcp37-64 master]# 
[root@dhcp37-64 master]# 
[root@dhcp37-64 master]# gluster volume geo-replication master 10.70.43.82::slave status
 
MASTER NODE     MASTER VOL    MASTER BRICK       SLAVE USER    SLAVE                 SLAVE NODE      STATUS     CRAWL STATUS     LAST_SYNCED                  
---------------------------------------------------------------------------------------------------------------------------------------------------
10.70.37.64     master        /rhs/brick1/b1     root          10.70.43.82::slave    10.70.42.39     Active     History Crawl    2017-08-20 21:00:34          
10.70.37.64     master        /rhs/brick2/b5     root          10.70.43.82::slave    10.70.42.39     Active     History Crawl    2017-08-20 21:01:35          
10.70.37.64     master        /rhs/brick3/b9     root          10.70.43.82::slave    10.70.42.39     Active     History Crawl    2017-08-20 21:06:36          
10.70.37.133    master        /rhs/brick1/b3     root          10.70.43.82::slave    10.70.43.82     Active     History Crawl    2017-08-20 21:11:23          
10.70.37.133    master        /rhs/brick2/b7     root          10.70.43.82::slave    10.70.43.82     Active     History Crawl    2017-08-20 21:11:24          
10.70.37.129    master        /rhs/brick1/b4     root          10.70.43.82::slave    10.70.43.101    Passive    N/A              N/A                          
10.70.37.129    master        /rhs/brick2/b8     root          10.70.43.82::slave    10.70.43.101    Passive    N/A              N/A                          
10.70.37.60     master        /rhs/brick1/b2     root          10.70.43.82::slave    10.70.43.190    Passive    N/A              N/A                          
10.70.37.60     master        /rhs/brick2/b6     root          10.70.43.82::slave    10.70.43.190    Passive    N/A              N/A                          
10.70.37.60     master        /rhs/brick3/b10    root          10.70.43.82::slave    10.70.43.190    Passive    N/A              N/A                          
[root@dhcp37-64 master]# 

Version-Release number of selected component (if applicable):
=============================================================

glusterfs-geo-replication-3.8.4-41.el7rhgs.x86_64



How reproducible:
=================

I have 2 systems with the same use case having more files at slave, I have confirmed this issue on one. Looking into the another system to confirm if it is the same issue. 

Additional info:
================

Changelog processed for one of the gfid:

[root@dhcp37-64 ssh%3A%2F%2Froot%4010.70.43.82%3Agluster%3A%2F%2F127.0.0.1%3Aslave]#  grep -ri "1bdb8a40-0abb-4471-9348-dae93fcb4528" *
Binary file 7276cecc9d436837c6dcbfda8f8e2d3b/.processed/archive_201708.tar matches
7276cecc9d436837c6dcbfda8f8e2d3b/.processed/CHANGELOG.1503243037:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 CREATE 33188 0 0 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f7a9%25%25RB7HJ9KRCT
7276cecc9d436837c6dcbfda8f8e2d3b/.processed/CHANGELOG.1503243037:D 1bdb8a40-0abb-4471-9348-dae93fcb4528
7276cecc9d436837c6dcbfda8f8e2d3b/.processed/CHANGELOG.1503243067:M 1bdb8a40-0abb-4471-9348-dae93fcb4528 SETATTR
7276cecc9d436837c6dcbfda8f8e2d3b/.processed/CHANGELOG.1503243097:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 LINK 93bfd3a2-358c-441b-87b9-d6a8a868be91%2F5998f7e3%25%257YY4EWS6EU
7276cecc9d436837c6dcbfda8f8e2d3b/.processed/CHANGELOG.1503243127:M 1bdb8a40-0abb-4471-9348-dae93fcb4528 SETATTR
7276cecc9d436837c6dcbfda8f8e2d3b/.processed/CHANGELOG.1503243187:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 LINK 93bfd3a2-358c-441b-87b9-d6a8a868be91%2F5998f842%25%25SO1LC2DHJQ
7276cecc9d436837c6dcbfda8f8e2d3b/.processed/CHANGELOG.1503243217:D 1bdb8a40-0abb-4471-9348-dae93fcb4528
7276cecc9d436837c6dcbfda8f8e2d3b/.processed/CHANGELOG.1503243247:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 LINK 93bfd3a2-358c-441b-87b9-d6a8a868be91%2F5998f876%25%2506ZOMZH50H
7276cecc9d436837c6dcbfda8f8e2d3b/.processed/CHANGELOG.1503243277:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 RENAME 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f7a9%25%25RB7HJ9KRCT 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f896%25%25HRXY4CF4LR
7276cecc9d436837c6dcbfda8f8e2d3b/.processed/CHANGELOG.1503243307:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 LINK 93bfd3a2-358c-441b-87b9-d6a8a868be91%2F5998f8b6%25%25J2NOW4HN8O
7276cecc9d436837c6dcbfda8f8e2d3b/.processed/CHANGELOG.1503243367:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 LINK 93bfd3a2-358c-441b-87b9-d6a8a868be91%2F5998f8ea%25%25N7FVXBFAP4
7276cecc9d436837c6dcbfda8f8e2d3b/.processed/CHANGELOG.1503243397:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 RENAME 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f896%25%25HRXY4CF4LR 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f908%25%25C7SK0C8YNY
Binary file 7276cecc9d436837c6dcbfda8f8e2d3b/.history/.processed/archive_201708.tar matches
7276cecc9d436837c6dcbfda8f8e2d3b/.history/.processed/CHANGELOG.1503243397:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 RENAME 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f896%25%25HRXY4CF4LR 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f908%25%25C7SK0C8YNY
7276cecc9d436837c6dcbfda8f8e2d3b/.history/.processed/CHANGELOG.1503243427:M 1bdb8a40-0abb-4471-9348-dae93fcb4528 SETATTR
7276cecc9d436837c6dcbfda8f8e2d3b/.history/.processed/CHANGELOG.1503243457:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 RENAME 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f908%25%25C7SK0C8YNY 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f947%25%25Y2TXWBTNNE
7276cecc9d436837c6dcbfda8f8e2d3b/.history/.processed/CHANGELOG.1503243517:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 LINK 93bfd3a2-358c-441b-87b9-d6a8a868be91%2F5998f987%25%25L3OBVJE1CI
7276cecc9d436837c6dcbfda8f8e2d3b/.history/.processed/CHANGELOG.1503245428:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 LINK 93bfd3a2-358c-441b-87b9-d6a8a868be91%2F5998f9af%25%25PSFQMMZPPY
7276cecc9d436837c6dcbfda8f8e2d3b/.history/.processed/CHANGELOG.1503245653:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 UNLINK 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f947%25%25Y2TXWBTNNE
7276cecc9d436837c6dcbfda8f8e2d3b/.history/.processed/CHANGELOG.1503245653:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 MKNOD 32769 0 0 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f9fd%25%250R22IFYPJS
Binary file c19b89ac45352ab8c894d210d136dd56/.history/.processed/archive_201708.tar matches
c19b89ac45352ab8c894d210d136dd56/.history/.processed/CHANGELOG.1503245608:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 MKNOD 33280 0 0 93bfd3a2-358c-441b-87b9-d6a8a868be91%2F5998f8b6%25%25J2NOW4HN8O
c19b89ac45352ab8c894d210d136dd56/.history/.processed/CHANGELOG.1503245923:D 1bdb8a40-0abb-4471-9348-dae93fcb4528
c19b89ac45352ab8c894d210d136dd56/.history/.processed/CHANGELOG.1503245923:M 1bdb8a40-0abb-4471-9348-dae93fcb4528 SETATTR
c19b89ac45352ab8c894d210d136dd56/.history/.processed/CHANGELOG.1503245923:M 1bdb8a40-0abb-4471-9348-dae93fcb4528 REMOVEXATTR
[root@dhcp37-64 ssh%3A%2F%2Froot%4010.70.43.82%3Agluster%3A%2F%2F127.0.0.1%3Aslave]# 








[root@dhcp37-60 ssh%3A%2F%2Froot%4010.70.43.82%3Agluster%3A%2F%2F127.0.0.1%3Aslave]#  grep -ri "1bdb8a40-0abb-4471-9348-dae93fcb4528" *
f450f0c1626707f4925dd9409a882047/.processed/CHANGELOG.1503243416:M 1bdb8a40-0abb-4471-9348-dae93fcb4528 SETATTR
f450f0c1626707f4925dd9409a882047/.processed/CHANGELOG.1503243446:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 RENAME 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f908%25%25C7SK0C8YNY 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f947%25%25Y2TXWBTNNE
Binary file f450f0c1626707f4925dd9409a882047/.processed/archive_201708.tar matches
f450f0c1626707f4925dd9409a882047/.processed/CHANGELOG.1503243041:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 CREATE 33188 0 0 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f7a9%25%25RB7HJ9KRCT
f450f0c1626707f4925dd9409a882047/.processed/CHANGELOG.1503243041:D 1bdb8a40-0abb-4471-9348-dae93fcb4528
f450f0c1626707f4925dd9409a882047/.processed/CHANGELOG.1503243071:M 1bdb8a40-0abb-4471-9348-dae93fcb4528 SETATTR
f450f0c1626707f4925dd9409a882047/.processed/CHANGELOG.1503243101:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 LINK 93bfd3a2-358c-441b-87b9-d6a8a868be91%2F5998f7e3%25%257YY4EWS6EU
f450f0c1626707f4925dd9409a882047/.processed/CHANGELOG.1503243131:M 1bdb8a40-0abb-4471-9348-dae93fcb4528 SETATTR
f450f0c1626707f4925dd9409a882047/.processed/CHANGELOG.1503243191:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 LINK 93bfd3a2-358c-441b-87b9-d6a8a868be91%2F5998f842%25%25SO1LC2DHJQ
f450f0c1626707f4925dd9409a882047/.processed/CHANGELOG.1503243221:D 1bdb8a40-0abb-4471-9348-dae93fcb4528
f450f0c1626707f4925dd9409a882047/.processed/CHANGELOG.1503243251:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 LINK 93bfd3a2-358c-441b-87b9-d6a8a868be91%2F5998f876%25%2506ZOMZH50H
f450f0c1626707f4925dd9409a882047/.processed/CHANGELOG.1503243281:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 RENAME 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f7a9%25%25RB7HJ9KRCT 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f896%25%25HRXY4CF4LR
f450f0c1626707f4925dd9409a882047/.processed/CHANGELOG.1503243311:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 LINK 93bfd3a2-358c-441b-87b9-d6a8a868be91%2F5998f8b6%25%25J2NOW4HN8O
f450f0c1626707f4925dd9409a882047/.processed/CHANGELOG.1503243356:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 LINK 93bfd3a2-358c-441b-87b9-d6a8a868be91%2F5998f8ea%25%25N7FVXBFAP4
f450f0c1626707f4925dd9409a882047/.processed/CHANGELOG.1503243386:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 RENAME 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f896%25%25HRXY4CF4LR 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f908%25%25C7SK0C8YNY
f450f0c1626707f4925dd9409a882047/.processed/CHANGELOG.1503243521:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 LINK 93bfd3a2-358c-441b-87b9-d6a8a868be91%2F5998f987%25%25L3OBVJE1CI
f450f0c1626707f4925dd9409a882047/.processed/CHANGELOG.1503243551:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 LINK 93bfd3a2-358c-441b-87b9-d6a8a868be91%2F5998f9af%25%25PSFQMMZPPY
f450f0c1626707f4925dd9409a882047/.processed/CHANGELOG.1503243641:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 RENAME 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f947%25%25Y2TXWBTNNE 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f9fd%25%250R22IFYPJS
Binary file f450f0c1626707f4925dd9409a882047/.history/.processed/archive_201708.tar matches
f450f0c1626707f4925dd9409a882047/.history/.processed/CHANGELOG.1503243041:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 CREATE 33188 0 0 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f7a9%25%25RB7HJ9KRCT
f450f0c1626707f4925dd9409a882047/.history/.processed/CHANGELOG.1503243041:D 1bdb8a40-0abb-4471-9348-dae93fcb4528
f450f0c1626707f4925dd9409a882047/.history/.processed/CHANGELOG.1503243071:M 1bdb8a40-0abb-4471-9348-dae93fcb4528 SETATTR
f450f0c1626707f4925dd9409a882047/.history/.processed/CHANGELOG.1503243101:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 LINK 93bfd3a2-358c-441b-87b9-d6a8a868be91%2F5998f7e3%25%257YY4EWS6EU
f450f0c1626707f4925dd9409a882047/.history/.processed/CHANGELOG.1503243131:M 1bdb8a40-0abb-4471-9348-dae93fcb4528 SETATTR
f450f0c1626707f4925dd9409a882047/.history/.processed/CHANGELOG.1503243191:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 LINK 93bfd3a2-358c-441b-87b9-d6a8a868be91%2F5998f842%25%25SO1LC2DHJQ
f450f0c1626707f4925dd9409a882047/.history/.processed/CHANGELOG.1503243221:D 1bdb8a40-0abb-4471-9348-dae93fcb4528
f450f0c1626707f4925dd9409a882047/.history/.processed/CHANGELOG.1503243251:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 LINK 93bfd3a2-358c-441b-87b9-d6a8a868be91%2F5998f876%25%2506ZOMZH50H
f450f0c1626707f4925dd9409a882047/.history/.processed/CHANGELOG.1503243281:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 RENAME 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f7a9%25%25RB7HJ9KRCT 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f896%25%25HRXY4CF4LR
f450f0c1626707f4925dd9409a882047/.history/.processed/CHANGELOG.1503243311:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 LINK 93bfd3a2-358c-441b-87b9-d6a8a868be91%2F5998f8b6%25%25J2NOW4HN8O
f450f0c1626707f4925dd9409a882047/.history/.processed/CHANGELOG.1503243356:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 LINK 93bfd3a2-358c-441b-87b9-d6a8a868be91%2F5998f8ea%25%25N7FVXBFAP4
f450f0c1626707f4925dd9409a882047/.history/.processed/CHANGELOG.1503243386:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 RENAME 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f896%25%25HRXY4CF4LR 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f908%25%25C7SK0C8YNY
f450f0c1626707f4925dd9409a882047/.history/.processed/CHANGELOG.1503243416:M 1bdb8a40-0abb-4471-9348-dae93fcb4528 SETATTR
f450f0c1626707f4925dd9409a882047/.history/.processed/CHANGELOG.1503243446:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 RENAME 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f908%25%25C7SK0C8YNY 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f947%25%25Y2TXWBTNNE
f450f0c1626707f4925dd9409a882047/.history/.processed/CHANGELOG.1503243521:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 LINK 93bfd3a2-358c-441b-87b9-d6a8a868be91%2F5998f987%25%25L3OBVJE1CI
f450f0c1626707f4925dd9409a882047/.history/.processed/CHANGELOG.1503243551:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 LINK 93bfd3a2-358c-441b-87b9-d6a8a868be91%2F5998f9af%25%25PSFQMMZPPY
f450f0c1626707f4925dd9409a882047/.history/.processed/CHANGELOG.1503243641:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 RENAME 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f947%25%25Y2TXWBTNNE 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f9fd%25%250R22IFYPJS
[root@dhcp37-60 ssh%3A%2F%2Froot%4010.70.43.82%3Agluster%3A%2F%2F127.0.0.1%3Aslave]#

Comment 26 errata-xmlrpc 2018-09-04 06:35:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:2607