| Summary: | [geo-rep]: Slave has more entries than Master in multiple hardlink/rename scenario | ||
|---|---|---|---|
| Product: | Red Hat Gluster Storage | Reporter: | Rahul Hinduja <rhinduja> |
| Component: | geo-replication | Assignee: | Kotresh HR <khiremat> |
| Status: | CLOSED ERRATA | QA Contact: | Rochelle <rallan> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | rhgs-3.3 | CC: | amukherj, apaladug, atumball, csaba, khiremat, rallan, rcyriac, rhinduja, rhs-bugs, sheggodu, storage-qa-internal |
| Target Milestone: | --- | Flags: | khiremat:
needinfo?
(apaladug) |
| Target Release: | RHGS 3.4.0 | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-09-04 06:35:11 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Bug Depends On: | 1599587 | ||
| Bug Blocks: | 1503134 | ||
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:2607 |
Description of problem: ======================= During a scenario where lot of hardlink and renames of same files were tried while one of the brick is down. The end results shows additional files at slave: Following are two addtional files at slave on a single directory: At Slave: --------- [root@dj ~]# ls /mnt/slave/f/kill-scenario/thread0/level01/level11/level21/level31/level41/level51/level61/level71/level81/level91/ | grep 5998f908%%C7SK0C8YNY 5998f908%%C7SK0C8YNY [root@dj ~]# ls /mnt/slave/f/kill-scenario/thread0/level01/level11/level21/level31/level41/level51/level61/level71/level81/level91/ | grep 5998f908%%DMB81F7NUA 5998f908%%DMB81F7NUA [root@dj ~]# At Master: ---------- [root@dj ~]# ls /mnt/master/f/kill-scenario/thread0/level01/level11/level21/level31/level41/level51/level61/level71/level81/level91/ | grep 5998f908%%C7SK0C8YNY [root@dj ~]# ls /mnt/master/f/kill-scenario/thread0/level01/level11/level21/level31/level41/level51/level61/level71/level81/level91/ | grep 5998f908%%DMB81F7NUA [root@dj ~]# Geo-Replication logs shows the Entry Failure for these files during rename as: [2017-08-20 16:28:52.920265] I [master(/rhs/brick1/b1):1203:crawl] _GMaster: slave's time: (1503243034, 0) [2017-08-20 16:28:52.987740] I [master(/rhs/brick2/b5):1203:crawl] _GMaster: slave's time: (1503243095, 0) [2017-08-20 16:29:16.379971] I [master(/rhs/brick3/b9):1203:crawl] _GMaster: slave's time: (1503243396, 0) [2017-08-20 16:31:56.771578] E [master(/rhs/brick3/b9):782:log_failures] _GMaster: ENTRY FAILED: ({'stat': {'atime': 1503243030.460567, 'gid': 0, 'mtime': 1503243210.3251548, 'mode': 32769, 'uid': 0}, 'entry1': '.gfid/16eb8f74-2295-4f18-8e82-64ed85999c81/5998f947%%Y2TXWBTNNE', 'gfid': '1bdb8a40-0abb-4471-9348-dae93fcb4528', 'link': None, 'entry': '.gfid/16eb8f74-2295-4f18-8e82-64ed85999c81/5998f908%%C7SK0C8YNY', 'op': 'RENAME'}, 2) [2017-08-20 16:31:56.774578] E [master(/rhs/brick3/b9):782:log_failures] _GMaster: ENTRY FAILED: ({'stat': {'atime': 1503243030.4875665, 'gid': 0, 'mtime': 1503243210.3361545, 'mode': 33161, 'uid': 0}, 'entry1': '.gfid/16eb8f74-2295-4f18-8e82-64ed85999c81/5998f947%%M3HN9AVSW9', 'gfid': 'b55ee2c7-0831-4f54-a462-cdbf77377aa6', 'link': None, 'entry': '.gfid/16eb8f74-2295-4f18-8e82-64ed85999c81/5998f908%%DMB81F7NUA', 'op': 'RENAME'}, 2) [2017-08-20 16:31:56.777143] E [master(/rhs/brick3/b9):782:log_failures] _GMaster: ENTRY FAILED: ({'stat': {'atime': 1503243030.6145642, 'gid': 0, 'mtime': 1503243210.461152, 'mode': 33242, 'uid': 0}, 'entry1': '.gfid/f2f823c5-a8df-44ff-839a-5aa6056fd47d/5998f948%%RSV6VIJG56', 'gfid': '7ac38ace-02ef-4312-9cd3-8f431df2d258', 'link': None, 'entry': '.gfid/f2f823c5-a8df-44ff-839a-5aa6056fd47d/5998f908%%15AWA4KG78', 'op': 'RENAME'}, 2) Initial State: ============== [root@dhcp37-64 master]# gluster volume geo-replication master 10.70.43.82::slave status MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED ----------------------------------------------------------------------------------------------------------------------------------------------------- 10.70.37.64 master /rhs/brick1/b1 root 10.70.43.82::slave 10.70.42.39 Active Changelog Crawl 2017-08-20 21:00:19 10.70.37.64 master /rhs/brick2/b5 root 10.70.43.82::slave 10.70.42.39 Active Changelog Crawl 2017-08-20 21:00:20 10.70.37.64 master /rhs/brick3/b9 root 10.70.43.82::slave 10.70.42.39 Active Changelog Crawl 2017-08-20 21:00:21 10.70.37.133 master /rhs/brick1/b3 root 10.70.43.82::slave 10.70.43.82 Active Changelog Crawl 2017-08-20 21:00:23 10.70.37.133 master /rhs/brick2/b7 root 10.70.43.82::slave 10.70.43.82 Active Changelog Crawl 2017-08-20 21:00:24 10.70.37.129 master /rhs/brick1/b4 root 10.70.43.82::slave 10.70.43.101 Passive N/A N/A 10.70.37.129 master /rhs/brick2/b8 root 10.70.43.82::slave 10.70.43.101 Passive N/A N/A 10.70.37.60 master /rhs/brick1/b2 root 10.70.43.82::slave 10.70.43.190 Passive N/A N/A 10.70.37.60 master /rhs/brick2/b6 root 10.70.43.82::slave 10.70.43.190 Passive N/A N/A 10.70.37.60 master /rhs/brick3/b10 root 10.70.43.82::slave 10.70.43.190 Passive N/A N/A [root@dhcp37-64 master]# Scenario: +++++++++ Start writing data at mount: ============================ for i in {create,chmod,hardlink,chgrp,symlink,hardlink,truncate,hardlink,rename,hardlink,symlink,hardlink,rename,chown,rename,create,hardlink,hardlink,symlink,rename}; do crefi --multi -n 5 -b 10 -d 10 --max=10K --min=500 --random -T 10 -t text --fop=$i /mnt/master/f/kill-scenario/ ; sleep 10 ; done Kill brick b1 => Wait for status to faulty and b2 to become Active (History Crawl) Kill brick b5 => Wait for status to faulty and b2 to become Active (History Crawl) Kill brick b9 => Wait for status to faulty and b10 to become Active (History Crawl) Data completed in between.. =========================== Stop the geo-replication... =========================== Bring back the offline bricks using start force: ================================================ Wait for heal to complete: ========================== [root@dhcp37-64 master]# gluster volume heal master info Brick 10.70.37.64:/rhs/brick1/b1 Status: Connected Number of entries: 0 Brick 10.70.37.60:/rhs/brick1/b2 Status: Connected Number of entries: 0 Brick 10.70.37.133:/rhs/brick1/b3 Status: Connected Number of entries: 0 Brick 10.70.37.129:/rhs/brick1/b4 Status: Connected Number of entries: 0 Brick 10.70.37.64:/rhs/brick2/b5 Status: Connected Number of entries: 0 Brick 10.70.37.60:/rhs/brick2/b6 Status: Connected Number of entries: 0 Brick 10.70.37.133:/rhs/brick2/b7 Status: Connected Number of entries: 0 Brick 10.70.37.129:/rhs/brick2/b8 Status: Connected Number of entries: 0 Brick 10.70.37.64:/rhs/brick3/b9 Status: Connected Number of entries: 0 Brick 10.70.37.60:/rhs/brick3/b10 Status: Connected Number of entries: 0 [root@dhcp37-64 master]# [root@dhcp37-64 master]# [root@dhcp37-64 master]# gluster volume status master Status of volume: master Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick 10.70.37.64:/rhs/brick1/b1 49156 0 Y 14161 Brick 10.70.37.60:/rhs/brick1/b2 49152 0 Y 13358 Brick 10.70.37.133:/rhs/brick1/b3 49152 0 Y 13371 Brick 10.70.37.129:/rhs/brick1/b4 49152 0 Y 13364 Brick 10.70.37.64:/rhs/brick2/b5 49157 0 Y 14182 Brick 10.70.37.60:/rhs/brick2/b6 49153 0 Y 13377 Brick 10.70.37.133:/rhs/brick2/b7 49153 0 Y 13390 Brick 10.70.37.129:/rhs/brick2/b8 49153 0 Y 13383 Brick 10.70.37.64:/rhs/brick3/b9 49158 0 Y 14203 Brick 10.70.37.60:/rhs/brick3/b10 49154 0 Y 13396 NFS Server on localhost 2049 0 Y 14226 Self-heal Daemon on localhost N/A N/A Y 14236 NFS Server on 10.70.37.133 2049 0 Y 4019 Self-heal Daemon on 10.70.37.133 N/A N/A Y 4028 NFS Server on 10.70.37.129 2049 0 Y 32227 Self-heal Daemon on 10.70.37.129 N/A N/A Y 32236 NFS Server on 10.70.37.60 2049 0 Y 1916 Self-heal Daemon on 10.70.37.60 N/A N/A Y 1925 Task Status of Volume master ------------------------------------------------------------------------------ There are no active volume tasks [root@dhcp37-64 master]# [root@dhcp37-64 master]# [root@dhcp37-64 master]# gluster volume geo-replication master 10.70.43.82::slave status MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED ---------------------------------------------------------------------------------------------------------------------------------------- 10.70.37.64 master /rhs/brick1/b1 root 10.70.43.82::slave N/A Stopped N/A N/A 10.70.37.64 master /rhs/brick2/b5 root 10.70.43.82::slave N/A Stopped N/A N/A 10.70.37.64 master /rhs/brick3/b9 root 10.70.43.82::slave N/A Stopped N/A N/A 10.70.37.133 master /rhs/brick1/b3 root 10.70.43.82::slave N/A Stopped N/A N/A 10.70.37.133 master /rhs/brick2/b7 root 10.70.43.82::slave N/A Stopped N/A N/A 10.70.37.129 master /rhs/brick1/b4 root 10.70.43.82::slave N/A Stopped N/A N/A 10.70.37.129 master /rhs/brick2/b8 root 10.70.43.82::slave N/A Stopped N/A N/A 10.70.37.60 master /rhs/brick1/b2 root 10.70.43.82::slave N/A Stopped N/A N/A 10.70.37.60 master /rhs/brick2/b6 root 10.70.43.82::slave N/A Stopped N/A N/A 10.70.37.60 master /rhs/brick3/b10 root 10.70.43.82::slave N/A Stopped N/A N/A Start the geo-replication: ========================== [root@dhcp37-64 master]# gluster volume geo-replication master 10.70.43.82::slave start Starting geo-replication session between master & 10.70.43.82::slave has been successful [root@dhcp37-64 master]# gluster volume geo-replication master 10.70.43.82::slave status MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED ------------------------------------------------------------------------------------------------------------------------------------------------ 10.70.37.64 master /rhs/brick1/b1 root 10.70.43.82::slave N/A Initializing... N/A N/A 10.70.37.64 master /rhs/brick2/b5 root 10.70.43.82::slave N/A Initializing... N/A N/A 10.70.37.64 master /rhs/brick3/b9 root 10.70.43.82::slave N/A Initializing... N/A N/A 10.70.37.129 master /rhs/brick1/b4 root 10.70.43.82::slave N/A Initializing... N/A N/A 10.70.37.129 master /rhs/brick2/b8 root 10.70.43.82::slave N/A Initializing... N/A N/A 10.70.37.133 master /rhs/brick1/b3 root 10.70.43.82::slave N/A Initializing... N/A N/A 10.70.37.133 master /rhs/brick2/b7 root 10.70.43.82::slave N/A Initializing... N/A N/A 10.70.37.60 master /rhs/brick1/b2 root 10.70.43.82::slave N/A Initializing... N/A N/A 10.70.37.60 master /rhs/brick2/b6 root 10.70.43.82::slave N/A Initializing... N/A N/A 10.70.37.60 master /rhs/brick3/b10 root 10.70.43.82::slave N/A Initializing... N/A N/A [root@dhcp37-64 master]# [root@dhcp37-64 master]# [root@dhcp37-64 master]# gluster volume geo-replication master 10.70.43.82::slave status MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED ------------------------------------------------------------------------------------------------------------------------------------------------ 10.70.37.64 master /rhs/brick1/b1 root 10.70.43.82::slave N/A Initializing... N/A N/A 10.70.37.64 master /rhs/brick2/b5 root 10.70.43.82::slave N/A Initializing... N/A N/A 10.70.37.64 master /rhs/brick3/b9 root 10.70.43.82::slave N/A Initializing... N/A N/A 10.70.37.133 master /rhs/brick1/b3 root 10.70.43.82::slave N/A Initializing... N/A N/A 10.70.37.133 master /rhs/brick2/b7 root 10.70.43.82::slave N/A Initializing... N/A N/A 10.70.37.129 master /rhs/brick1/b4 root 10.70.43.82::slave N/A Initializing... N/A N/A 10.70.37.129 master /rhs/brick2/b8 root 10.70.43.82::slave N/A Initializing... N/A N/A 10.70.37.60 master /rhs/brick1/b2 root 10.70.43.82::slave N/A Initializing... N/A N/A 10.70.37.60 master /rhs/brick2/b6 root 10.70.43.82::slave N/A Initializing... N/A N/A 10.70.37.60 master /rhs/brick3/b10 root 10.70.43.82::slave N/A Initializing... N/A N/A [root@dhcp37-64 master]# [root@dhcp37-64 master]# [root@dhcp37-64 master]# [root@dhcp37-64 master]# gluster volume geo-replication master 10.70.43.82::slave status MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED --------------------------------------------------------------------------------------------------------------------------------------------------- 10.70.37.64 master /rhs/brick1/b1 root 10.70.43.82::slave 10.70.42.39 Active History Crawl 2017-08-20 21:00:34 10.70.37.64 master /rhs/brick2/b5 root 10.70.43.82::slave 10.70.42.39 Active History Crawl 2017-08-20 21:01:35 10.70.37.64 master /rhs/brick3/b9 root 10.70.43.82::slave 10.70.42.39 Active History Crawl 2017-08-20 21:06:36 10.70.37.133 master /rhs/brick1/b3 root 10.70.43.82::slave 10.70.43.82 Active History Crawl 2017-08-20 21:11:23 10.70.37.133 master /rhs/brick2/b7 root 10.70.43.82::slave 10.70.43.82 Active History Crawl 2017-08-20 21:11:24 10.70.37.129 master /rhs/brick1/b4 root 10.70.43.82::slave 10.70.43.101 Passive N/A N/A 10.70.37.129 master /rhs/brick2/b8 root 10.70.43.82::slave 10.70.43.101 Passive N/A N/A 10.70.37.60 master /rhs/brick1/b2 root 10.70.43.82::slave 10.70.43.190 Passive N/A N/A 10.70.37.60 master /rhs/brick2/b6 root 10.70.43.82::slave 10.70.43.190 Passive N/A N/A 10.70.37.60 master /rhs/brick3/b10 root 10.70.43.82::slave 10.70.43.190 Passive N/A N/A [root@dhcp37-64 master]# Version-Release number of selected component (if applicable): ============================================================= glusterfs-geo-replication-3.8.4-41.el7rhgs.x86_64 How reproducible: ================= I have 2 systems with the same use case having more files at slave, I have confirmed this issue on one. Looking into the another system to confirm if it is the same issue. Additional info: ================ Changelog processed for one of the gfid: [root@dhcp37-64 ssh%3A%2F%2Froot%4010.70.43.82%3Agluster%3A%2F%2F127.0.0.1%3Aslave]# grep -ri "1bdb8a40-0abb-4471-9348-dae93fcb4528" * Binary file 7276cecc9d436837c6dcbfda8f8e2d3b/.processed/archive_201708.tar matches 7276cecc9d436837c6dcbfda8f8e2d3b/.processed/CHANGELOG.1503243037:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 CREATE 33188 0 0 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f7a9%25%25RB7HJ9KRCT 7276cecc9d436837c6dcbfda8f8e2d3b/.processed/CHANGELOG.1503243037:D 1bdb8a40-0abb-4471-9348-dae93fcb4528 7276cecc9d436837c6dcbfda8f8e2d3b/.processed/CHANGELOG.1503243067:M 1bdb8a40-0abb-4471-9348-dae93fcb4528 SETATTR 7276cecc9d436837c6dcbfda8f8e2d3b/.processed/CHANGELOG.1503243097:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 LINK 93bfd3a2-358c-441b-87b9-d6a8a868be91%2F5998f7e3%25%257YY4EWS6EU 7276cecc9d436837c6dcbfda8f8e2d3b/.processed/CHANGELOG.1503243127:M 1bdb8a40-0abb-4471-9348-dae93fcb4528 SETATTR 7276cecc9d436837c6dcbfda8f8e2d3b/.processed/CHANGELOG.1503243187:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 LINK 93bfd3a2-358c-441b-87b9-d6a8a868be91%2F5998f842%25%25SO1LC2DHJQ 7276cecc9d436837c6dcbfda8f8e2d3b/.processed/CHANGELOG.1503243217:D 1bdb8a40-0abb-4471-9348-dae93fcb4528 7276cecc9d436837c6dcbfda8f8e2d3b/.processed/CHANGELOG.1503243247:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 LINK 93bfd3a2-358c-441b-87b9-d6a8a868be91%2F5998f876%25%2506ZOMZH50H 7276cecc9d436837c6dcbfda8f8e2d3b/.processed/CHANGELOG.1503243277:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 RENAME 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f7a9%25%25RB7HJ9KRCT 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f896%25%25HRXY4CF4LR 7276cecc9d436837c6dcbfda8f8e2d3b/.processed/CHANGELOG.1503243307:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 LINK 93bfd3a2-358c-441b-87b9-d6a8a868be91%2F5998f8b6%25%25J2NOW4HN8O 7276cecc9d436837c6dcbfda8f8e2d3b/.processed/CHANGELOG.1503243367:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 LINK 93bfd3a2-358c-441b-87b9-d6a8a868be91%2F5998f8ea%25%25N7FVXBFAP4 7276cecc9d436837c6dcbfda8f8e2d3b/.processed/CHANGELOG.1503243397:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 RENAME 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f896%25%25HRXY4CF4LR 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f908%25%25C7SK0C8YNY Binary file 7276cecc9d436837c6dcbfda8f8e2d3b/.history/.processed/archive_201708.tar matches 7276cecc9d436837c6dcbfda8f8e2d3b/.history/.processed/CHANGELOG.1503243397:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 RENAME 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f896%25%25HRXY4CF4LR 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f908%25%25C7SK0C8YNY 7276cecc9d436837c6dcbfda8f8e2d3b/.history/.processed/CHANGELOG.1503243427:M 1bdb8a40-0abb-4471-9348-dae93fcb4528 SETATTR 7276cecc9d436837c6dcbfda8f8e2d3b/.history/.processed/CHANGELOG.1503243457:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 RENAME 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f908%25%25C7SK0C8YNY 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f947%25%25Y2TXWBTNNE 7276cecc9d436837c6dcbfda8f8e2d3b/.history/.processed/CHANGELOG.1503243517:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 LINK 93bfd3a2-358c-441b-87b9-d6a8a868be91%2F5998f987%25%25L3OBVJE1CI 7276cecc9d436837c6dcbfda8f8e2d3b/.history/.processed/CHANGELOG.1503245428:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 LINK 93bfd3a2-358c-441b-87b9-d6a8a868be91%2F5998f9af%25%25PSFQMMZPPY 7276cecc9d436837c6dcbfda8f8e2d3b/.history/.processed/CHANGELOG.1503245653:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 UNLINK 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f947%25%25Y2TXWBTNNE 7276cecc9d436837c6dcbfda8f8e2d3b/.history/.processed/CHANGELOG.1503245653:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 MKNOD 32769 0 0 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f9fd%25%250R22IFYPJS Binary file c19b89ac45352ab8c894d210d136dd56/.history/.processed/archive_201708.tar matches c19b89ac45352ab8c894d210d136dd56/.history/.processed/CHANGELOG.1503245608:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 MKNOD 33280 0 0 93bfd3a2-358c-441b-87b9-d6a8a868be91%2F5998f8b6%25%25J2NOW4HN8O c19b89ac45352ab8c894d210d136dd56/.history/.processed/CHANGELOG.1503245923:D 1bdb8a40-0abb-4471-9348-dae93fcb4528 c19b89ac45352ab8c894d210d136dd56/.history/.processed/CHANGELOG.1503245923:M 1bdb8a40-0abb-4471-9348-dae93fcb4528 SETATTR c19b89ac45352ab8c894d210d136dd56/.history/.processed/CHANGELOG.1503245923:M 1bdb8a40-0abb-4471-9348-dae93fcb4528 REMOVEXATTR [root@dhcp37-64 ssh%3A%2F%2Froot%4010.70.43.82%3Agluster%3A%2F%2F127.0.0.1%3Aslave]# [root@dhcp37-60 ssh%3A%2F%2Froot%4010.70.43.82%3Agluster%3A%2F%2F127.0.0.1%3Aslave]# grep -ri "1bdb8a40-0abb-4471-9348-dae93fcb4528" * f450f0c1626707f4925dd9409a882047/.processed/CHANGELOG.1503243416:M 1bdb8a40-0abb-4471-9348-dae93fcb4528 SETATTR f450f0c1626707f4925dd9409a882047/.processed/CHANGELOG.1503243446:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 RENAME 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f908%25%25C7SK0C8YNY 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f947%25%25Y2TXWBTNNE Binary file f450f0c1626707f4925dd9409a882047/.processed/archive_201708.tar matches f450f0c1626707f4925dd9409a882047/.processed/CHANGELOG.1503243041:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 CREATE 33188 0 0 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f7a9%25%25RB7HJ9KRCT f450f0c1626707f4925dd9409a882047/.processed/CHANGELOG.1503243041:D 1bdb8a40-0abb-4471-9348-dae93fcb4528 f450f0c1626707f4925dd9409a882047/.processed/CHANGELOG.1503243071:M 1bdb8a40-0abb-4471-9348-dae93fcb4528 SETATTR f450f0c1626707f4925dd9409a882047/.processed/CHANGELOG.1503243101:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 LINK 93bfd3a2-358c-441b-87b9-d6a8a868be91%2F5998f7e3%25%257YY4EWS6EU f450f0c1626707f4925dd9409a882047/.processed/CHANGELOG.1503243131:M 1bdb8a40-0abb-4471-9348-dae93fcb4528 SETATTR f450f0c1626707f4925dd9409a882047/.processed/CHANGELOG.1503243191:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 LINK 93bfd3a2-358c-441b-87b9-d6a8a868be91%2F5998f842%25%25SO1LC2DHJQ f450f0c1626707f4925dd9409a882047/.processed/CHANGELOG.1503243221:D 1bdb8a40-0abb-4471-9348-dae93fcb4528 f450f0c1626707f4925dd9409a882047/.processed/CHANGELOG.1503243251:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 LINK 93bfd3a2-358c-441b-87b9-d6a8a868be91%2F5998f876%25%2506ZOMZH50H f450f0c1626707f4925dd9409a882047/.processed/CHANGELOG.1503243281:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 RENAME 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f7a9%25%25RB7HJ9KRCT 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f896%25%25HRXY4CF4LR f450f0c1626707f4925dd9409a882047/.processed/CHANGELOG.1503243311:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 LINK 93bfd3a2-358c-441b-87b9-d6a8a868be91%2F5998f8b6%25%25J2NOW4HN8O f450f0c1626707f4925dd9409a882047/.processed/CHANGELOG.1503243356:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 LINK 93bfd3a2-358c-441b-87b9-d6a8a868be91%2F5998f8ea%25%25N7FVXBFAP4 f450f0c1626707f4925dd9409a882047/.processed/CHANGELOG.1503243386:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 RENAME 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f896%25%25HRXY4CF4LR 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f908%25%25C7SK0C8YNY f450f0c1626707f4925dd9409a882047/.processed/CHANGELOG.1503243521:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 LINK 93bfd3a2-358c-441b-87b9-d6a8a868be91%2F5998f987%25%25L3OBVJE1CI f450f0c1626707f4925dd9409a882047/.processed/CHANGELOG.1503243551:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 LINK 93bfd3a2-358c-441b-87b9-d6a8a868be91%2F5998f9af%25%25PSFQMMZPPY f450f0c1626707f4925dd9409a882047/.processed/CHANGELOG.1503243641:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 RENAME 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f947%25%25Y2TXWBTNNE 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f9fd%25%250R22IFYPJS Binary file f450f0c1626707f4925dd9409a882047/.history/.processed/archive_201708.tar matches f450f0c1626707f4925dd9409a882047/.history/.processed/CHANGELOG.1503243041:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 CREATE 33188 0 0 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f7a9%25%25RB7HJ9KRCT f450f0c1626707f4925dd9409a882047/.history/.processed/CHANGELOG.1503243041:D 1bdb8a40-0abb-4471-9348-dae93fcb4528 f450f0c1626707f4925dd9409a882047/.history/.processed/CHANGELOG.1503243071:M 1bdb8a40-0abb-4471-9348-dae93fcb4528 SETATTR f450f0c1626707f4925dd9409a882047/.history/.processed/CHANGELOG.1503243101:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 LINK 93bfd3a2-358c-441b-87b9-d6a8a868be91%2F5998f7e3%25%257YY4EWS6EU f450f0c1626707f4925dd9409a882047/.history/.processed/CHANGELOG.1503243131:M 1bdb8a40-0abb-4471-9348-dae93fcb4528 SETATTR f450f0c1626707f4925dd9409a882047/.history/.processed/CHANGELOG.1503243191:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 LINK 93bfd3a2-358c-441b-87b9-d6a8a868be91%2F5998f842%25%25SO1LC2DHJQ f450f0c1626707f4925dd9409a882047/.history/.processed/CHANGELOG.1503243221:D 1bdb8a40-0abb-4471-9348-dae93fcb4528 f450f0c1626707f4925dd9409a882047/.history/.processed/CHANGELOG.1503243251:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 LINK 93bfd3a2-358c-441b-87b9-d6a8a868be91%2F5998f876%25%2506ZOMZH50H f450f0c1626707f4925dd9409a882047/.history/.processed/CHANGELOG.1503243281:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 RENAME 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f7a9%25%25RB7HJ9KRCT 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f896%25%25HRXY4CF4LR f450f0c1626707f4925dd9409a882047/.history/.processed/CHANGELOG.1503243311:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 LINK 93bfd3a2-358c-441b-87b9-d6a8a868be91%2F5998f8b6%25%25J2NOW4HN8O f450f0c1626707f4925dd9409a882047/.history/.processed/CHANGELOG.1503243356:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 LINK 93bfd3a2-358c-441b-87b9-d6a8a868be91%2F5998f8ea%25%25N7FVXBFAP4 f450f0c1626707f4925dd9409a882047/.history/.processed/CHANGELOG.1503243386:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 RENAME 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f896%25%25HRXY4CF4LR 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f908%25%25C7SK0C8YNY f450f0c1626707f4925dd9409a882047/.history/.processed/CHANGELOG.1503243416:M 1bdb8a40-0abb-4471-9348-dae93fcb4528 SETATTR f450f0c1626707f4925dd9409a882047/.history/.processed/CHANGELOG.1503243446:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 RENAME 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f908%25%25C7SK0C8YNY 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f947%25%25Y2TXWBTNNE f450f0c1626707f4925dd9409a882047/.history/.processed/CHANGELOG.1503243521:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 LINK 93bfd3a2-358c-441b-87b9-d6a8a868be91%2F5998f987%25%25L3OBVJE1CI f450f0c1626707f4925dd9409a882047/.history/.processed/CHANGELOG.1503243551:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 LINK 93bfd3a2-358c-441b-87b9-d6a8a868be91%2F5998f9af%25%25PSFQMMZPPY f450f0c1626707f4925dd9409a882047/.history/.processed/CHANGELOG.1503243641:E 1bdb8a40-0abb-4471-9348-dae93fcb4528 RENAME 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f947%25%25Y2TXWBTNNE 16eb8f74-2295-4f18-8e82-64ed85999c81%2F5998f9fd%25%250R22IFYPJS [root@dhcp37-60 ssh%3A%2F%2Froot%4010.70.43.82%3Agluster%3A%2F%2F127.0.0.1%3Aslave]#