1025964 – dist-geo-rep: status details incorrectly indicate few files as skipped when all the files are properly synced to slave

Bug 1025964 - dist-geo-rep: status details incorrectly indicate few files as skipped when all the files are properly synced to slave

Summary: dist-geo-rep: status details incorrectly indicate few files as skipped when a...

Keywords:
Status:	CLOSED EOL
Alias:	None
Product:	Red Hat Gluster Storage
Classification:	Red Hat Storage
Component:	geo-replication
Sub Component:
Version:	2.1
Hardware:	x86_64
OS:	Linux
Priority:	medium
Severity:	medium
Target Milestone:	---
Target Release:	---
Assignee:	Bug Updates Notification Mailing List
QA Contact:	storage-qa-internal@redhat.com
Docs Contact:
URL:
Whiteboard:	status
Depends On:
Blocks:	1285203
TreeView+	depends on / blocked

Reported:	2013-11-02 13:39 UTC by M S Vishwanath Bhat
Modified:	2016-06-01 01:57 UTC (History)
CC List:	5 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:
Clone Of:
Clones:	1285203 (view as bug list)
Environment:
Last Closed:
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Description M S Vishwanath Bhat 2013-11-02 13:39:29 UTC

Description of problem:
The good news is that all the files are properly synced to slave. Thier arequl-checksums match. But status detail indicate that few files are skipped from syncing in the "TOTAL FILES SKIPPED" column. And even the log files indicate the same by logging the GFID of the file as skipped.

Version-Release number of selected component (if applicable):
glusterfs-3.4.0.38rhs-1.el6rhs.x86_64

How reproducible:
Not sure why some files were marked as skipped. So not sure how reproducible either.

Steps to Reproduce:
1. 
2.
3.

Actual results:
                                                   MASTER: master  SLAVE: falcon::slave
 
NODE                         HEALTH    UPTIME      FILES SYNCD    FILES PENDING    BYTES PENDING    DELETES PENDING    TOTAL FILES SKIPPED   
------------------------------------------------------------------------------------------------------------------------------------------
spitfire.blr.redhat.com      Stable    02:59:41    6240           0                0Bytes           0                  0                     
typhoon.blr.redhat.com       Stable    02:59:37    5852           0                0Bytes           0                  0                     
mustang.blr.redhat.com       Stable    02:59:37    0              0                0Bytes           0                  0                     
harrier.blr.redhat.com       Stable    00:35:49    10255          0                0Bytes           0                  68                    

harrier has skipped few files while syncing. But all the files are synced and their checksum matches.

[root@interceptor ~]# /opt/qa/tools/arequal-checksum /mnt/slave/

Entry counts
Regular files   : 10012
Directories     : 186
Symbolic links  : 0
Other           : 0
Total           : 10198

Metadata checksums
Regular files   : 3e9
Directories     : 3e9
Symbolic links  : 3e9
Other           : 3e9

Checksums
Regular files   : e8f5af99c3abeabff0322514757f02be
Directories     : 2e11310e000f445e
Symbolic links  : 0
Other           : 0
Total           : 36d6bb83b6dbac5f


[root@lightning smallfile]# /opt/qa/tools/arequal-checksum /mnt/master/

Entry counts
Regular files   : 10012
Directories     : 186
Symbolic links  : 0
Other           : 0
Total           : 10198

Metadata checksums
Regular files   : 3e9
Directories     : 3e9
Symbolic links  : 3e9
Other           : 3e9

Checksums
Regular files   : e8f5af99c3abeabff0322514757f02be
Directories     : 2e11310e000f445e
Symbolic links  : 0
Other           : 0
Total           : 36d6bb83b6dbac5f


Even the log files have relavant skipped logs.

[2013-11-02 17:23:28.616364] W [master(/rhs/bricks/brick2):785:process] _GMaster: SKIPPED GFID = eca046b7-69f4-4ea7-aff0-8414ec9dc825
[2013-11-02 17:23:28.616578] W [master(/rhs/bricks/brick2):785:process] _GMaster: SKIPPED GFID = 5022a9b1-a943-4423-8006-1100085774e2
[2013-11-02 17:23:28.616813] W [master(/rhs/bricks/brick2):785:process] _GMaster: SKIPPED GFID = b05c8000-7c60-4170-85a5-a8cb9b34cd78
[2013-11-02 17:23:28.617026] W [master(/rhs/bricks/brick2):785:process] _GMaster: SKIPPED GFID = 43727990-ea46-43ff-b94c-a9e1e1602a07
[2013-11-02 17:23:28.617367] W [master(/rhs/bricks/brick2):785:process] _GMaster: SKIPPED GFID = 55021faa-071c-458f-8ddb-551570aab995
[2013-11-02 17:23:28.617588] W [master(/rhs/bricks/brick2):785:process] _GMaster: SKIPPED GFID = 3cff1091-e71f-4fd2-b930-e3949044a0a1
[2013-11-02 17:23:28.617830] W [master(/rhs/bricks/brick2):785:process] _GMaster: SKIPPED GFID = 2b24ea19-2e98-46b1-a9d1-64e4e6c7f007
[2013-11-02 17:23:28.618043] W [master(/rhs/bricks/brick2):785:process] _GMaster: SKIPPED GFID = 26423c91-6a11-4bbc-a98e-fbebd64b9e07
[2013-11-02 17:23:28.618321] W [master(/rhs/bricks/brick2):785:process] _GMaster: SKIPPED GFID = f8bf308f-5649-45d5-8684-91ad53b6d988
[2013-11-02 17:23:28.618536] W [master(/rhs/bricks/brick2):785:process] _GMaster: SKIPPED GFID = b484958e-e1b2-4ad0-a88b-52c3866c2f10
[2013-11-02 17:23:28.618774] W [master(/rhs/bricks/brick2):785:process] _GMaster: SKIPPED GFID = cc430a10-5cd8-4b19-841a-db00c2e4ea57
[2013-11-02 17:23:28.618987] W [master(/rhs/bricks/brick2):785:process] _GMaster: SKIPPED GFID = c2e3aac7-7b76-4de3-a44d-a8e5863b52b9
[2013-11-02 17:23:28.619246] W [master(/rhs/bricks/brick2):785:process] _GMaster: SKIPPED GFID = 4b93c3af-3072-4e5a-9e88-c2c64b8f527d
[2013-11-02 17:23:28.619460] W [master(/rhs/bricks/brick2):785:process] _GMaster: SKIPPED GFID = 7cf4e9d7-4491-4cce-b2b3-02cb81487582
[2013-11-02 17:23:28.619693] W [master(/rhs/bricks/brick2):785:process] _GMaster: SKIPPED GFID = f87587a7-3c1e-4d78-b31d-e2e6e523c4cb
[2013-11-02 17:23:28.619906] W [master(/rhs/bricks/brick2):785:process] _GMaster: SKIPPED GFID = b58cc6ca-1f13-42a1-a38a-75dfee65c83e
[2013-11-02 17:23:28.620157] W [master(/rhs/bricks/brick2):785:process] _GMaster: SKIPPED GFID = 9dfec7d3-7474-4382-a2bb-71845eccaaf7
[2013-11-02 17:23:28.620379] W [master(/rhs/bricks/brick2):785:process] _GMaster: SKIPPED GFID = 79b6dfa8-3fb6-454a-8800-014ee605ac4a
[2013-11-02 17:23:28.620592] W [master(/rhs/bricks/brick2):785:process] _GMaster: SKIPPED GFID = 95fbbe92-7eb3-402a-8b3c-a3847fd6bfc9
[2013-11-02 17:23:28.620829] W [master(/rhs/bricks/brick2):785:process] _GMaster: SKIPPED GFID = efc59766-709c-4d57-a841-8bd7912d7e96
[2013-11-02 17:23:28.621041] W [master(/rhs/bricks/brick2):785:process] _GMaster: SKIPPED GFID = 809129d5-f4bb-4765-9c39-a9715d2a6139
[2013-11-02 17:23:28.621306] W [master(/rhs/bricks/brick2):785:process] _GMaster: SKIPPED GFID = 07e3845b-bd9d-4673-9aff-768acc8e9a38

Expected results:
When all the files are synced, status should not show any stale data of files being skipped.

Additional info:
Not sure if I can reproduce this once more. But will archive the logs.

Comment 2 Vijaykumar Koppad 2013-12-02 12:26:44 UTC

I saw this behavior with the build glusterfs-3.4.0.45geo-1 also.


[root@redcell ~]# gluster v geo master 10.70.43.76::slave status detail

MASTER NODE                MASTER VOL    MASTER BRICK             SLAVE                  STATUS     CHECKPOINT STATUS    CRAWL STATUS       FILES SYNCD    FILES PENDING    BYTES PENDING    DELETES PENDING    FILES SKIPPED
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
redcell.blr.redhat.com     master        /bricks/master_brick1    10.70.43.174::slave    Active     N/A                  Changelog Crawl    -3002          0                0                0                  3621
redlake.blr.redhat.com     master        /bricks/master_brick3    10.70.43.135::slave    Active     N/A                  Changelog Crawl    -2781          0                0                0                  3344
redcloak.blr.redhat.com    master        /bricks/master_brick2    10.70.43.76::slave     Passive    N/A                  N/A                0              0                0                0                  0
redwood.blr.redhat.com     master        /bricks/master_brick4    10.70.42.151::slave    Passive    N/A                  N/A                0              0                0                0                  0


It has shown some 6K skipped files out of 10K files, but all the files are synced and also checksum matches.

Comment 3 Aravinda VK 2015-11-25 08:50:38 UTC

Closing this bug since RHGS 2.1 release reached EOL. Required bugs are cloned to RHGS 3.1. Please re-open this issue if found again.

Comment 4 Aravinda VK 2015-11-25 08:51:58 UTC

Closing this bug since RHGS 2.1 release reached EOL. Required bugs are cloned to RHGS 3.1. Please re-open this issue if found again.

Note You need to log in before you can comment on or make changes to this bug.