Bug 1102550
| Summary: | Dist-geo-rep : geo-rep failed to sync some files, with error "(Operation not permitted)" on slaves. | |||
|---|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Vijaykumar Koppad <vkoppad> | |
| Component: | geo-replication | Assignee: | Aravinda VK <avishwan> | |
| Status: | CLOSED ERRATA | QA Contact: | Bhaskar Bandari <bbandari> | |
| Severity: | urgent | Docs Contact: | ||
| Priority: | high | |||
| Version: | rhgs-3.0 | CC: | aavati, avishwan, bbandari, csaba, david.macdonald, khiremat, nlevinki, nsathyan, rhs-bugs, ssamanta, vagarwal, vshankar | |
| Target Milestone: | --- | Keywords: | TestBlocker | |
| Target Release: | RHGS 3.0.0 | |||
| Hardware: | x86_64 | |||
| OS: | Linux | |||
| Whiteboard: | ||||
| Fixed In Version: | glusterfs-3.6.0.18-1 | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1105083 (view as bug list) | Environment: | ||
| Last Closed: | 2014-09-22 19:39:41 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | 1030309, 1286208 | |||
| Bug Blocks: | 1105083 | |||
|
Description
Vijaykumar Koppad
2014-05-29 07:55:58 UTC
"Operation not permitted" is due to the creation of file (using .gfid/<gfid>) in the normal code path. This happens when rsync triggers a file creation when the entry isn not found. But the entry should have already been there (as a part of entry_ops). This mostly looks like the same case where DHT was failing to create files (parallel MKDIR + CREATE). Vijaykumar, Were there any DHT related logs in the client mount apart from what is already present in comment #1? No, there were no DHT logs apart form what I have pasted there. Those logs are effect of something, doesn't look like the cause. If you check afr logs, all sub-volume returned with no gfid. I feel its related to the Bug 1102747. This is reproducible even with plane distribute volume on slave. In this case, the gfid of the directories are corrupted, and all the files under that are not created. gfid of one directory under which all files failed to sync >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on slave # getfattr -n glusterfs.gfid.string /mnt/slave/thread2/level07/level17/level27/level37/ getfattr: Removing leading '/' from absolute path names # file: mnt/slave/thread2/level07/level17/level27/level37/ glusterfs.gfid.string="00000000-0000-0000-5cf6-0b0100000000" on master # getfattr -n glusterfs.gfid.string /mnt/master/thread2/level07/level17/level27/level37/ getfattr: Removing leading '/' from absolute path names # file: mnt/master/thread2/level07/level17/level27/level37/ glusterfs.gfid.string="d0575c6a-b7c4-4b28-bfc0-2ddf57d35cb4 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and for the above gfid, the aux-mount logs have >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [2014-06-03 11:45:58.738230] W [fuse-bridge.c:1237:fuse_err_cbk] 0-glusterfs-fuse: 28187: SETXATTR() /.gfid/02a62fe6-6921-44a3-842a-189ad633b339 => -1 (No such file or directory) [2014-06-03 11:45:58.739845] W [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-slave-client-1: remote operation failed: File exists. Path: <gfid:fefa8327-f116-4768-9d8e-4395a727545d>/level55 [2014-06-03 11:45:58.740344] W [fuse-bridge.c:1237:fuse_err_cbk] 0-glusterfs-fuse: 22901: SETXATTR() /.gfid/fefa8327-f116-4768-9d8e-4395a727545d => -1 (File exists) [2014-06-03 11:45:58.750753] I [dht-layout.c:653:dht_layout_normalize] 0-slave-dht: found anomalies in (null). holes=1 overlaps=0 [2014-06-03 11:45:58.750995] W [dht-layout.c:178:dht_layout_search] 0-slave-dht: no subvolume for hash (value) = 2681045703 [2014-06-03 11:45:58.751033] W [fuse-bridge.c:1237:fuse_err_cbk] 0-glusterfs-fuse: 28190: SETXATTR() /.gfid/d0575c6a-b7c4-4b28-bfc0-2ddf57d35cb4 => -1 (No such file or directory) [2014-06-03 11:45:58.751635] W [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-slave-client-4: remote operation failed: File exists. Path: <gfid:02a62fe6-6921-44a3-842a-189ad633b339>/level47 [2014-06-03 11:45:58.751688] W [fuse-bridge.c:1237:fuse_err_cbk] 0-glusterfs-fuse: 28191: SETXATTR() /.gfid/02a62fe6-6921-44a3-842a-189ad633b339 => -1 (File exists) [2014-06-03 11:45:58.752223] W [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-slave-client-4: remote operation failed: File exists. Path: <gfid:d3e980b9-16b6-4a30-90f5-026efaefd1b9>/level57 [2014-06-03 11:45:58.752263] W [fuse-bridge.c:1237:fuse_err_cbk] 0-glusterfs-fuse: 28192: SETXATTR() /.gfid/d3e980b9-16b6-4a30-90f5-026efaefd1b9 => -1 (File exists) [2014-06-03 11:45:58.752802] W [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-slave-client-5: remote operation failed: File exists. Path: <gfid:fd054c2f-d892-4a40-a8b3-fb03a5b2d4e4>/level37 [2014-06-03 11:45:58.752840] W [fuse-bridge.c:1237:fuse_err_cbk] 0-glusterfs-fuse: 28193: SETXATTR() /.gfid/fd054c2f-d892-4a40-a8b3-fb03a5b2d4e4 => -1 (File exists) [2014-06-03 11:45:58.784464] W [dht-layout.c:178:dht_layout_search] 0-slave-dht: no subvolume for hash (value) = 577205183 [2014-06-03 11:45:58.784530] W [fuse-bridge.c:1237:fuse_err_cbk] 0-glusterfs-fuse: 28196: SETXATTR() /.gfid/efce422b-bc92-4615-94b9-a18a78c413fd => -1 (No such file or directory) [2014-06-03 11:45:58.784693] W [dht-layout.c:178:dht_layout_search] 0-slave-dht: no subvolume for hash (value) = 1812423478 [2014-06-03 11:45:58.784726] W [fuse-bridge.c:1237:fuse_err_cbk] 0-glusterfs-fuse: 28197: SETXATTR() /.gfid/4a354f13-e523-4c6d-ad2d-d2f562a87083 => -1 (Invalid argument) [2014-06-03 11:45:58.785022] W [dht-layout.c:178:dht_layout_search] 0-slave-dht: no subvolume for hash (value) = 1812423478 [2014-06-03 11:45:58.785058] W [fuse-bridge.c:1237:fuse_err_cbk] 0-glusterfs-fuse: 28198: SETXATTR() /.gfid/d0575c6a-b7c4-4b28-bfc0-2ddf57d35cb4 => -1 (Invalid argument) [2014-06-03 11:45:58.786243] W [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-slave-client-4: remote operation failed: File exists. Path: <gfid:ba0c0e46-a2cc-4a3c-8647-2f8cb3c7a78c>/level55 [2014-06-03 11:45:58.786285] W [fuse-bridge.c:1237:fuse_err_cbk] 0-glusterfs-fuse: 22953: SETXATTR() /.gfid/ba0c0e46-a2cc-4a3c-8647-2f8cb3c7a78c => -1 (File exists) [2014-06-03 11:45:58.798830] W [client-rpc-fops.c:306:client3_3_mkdir_cbk] 0-slave-client-5: remote operation failed: File exists. Path: <gfid:826f2fc5-ddf1-4eb2-88c8-8ad10f4ead14>/level55 [2014-06-03 11:45:58.798871] W [fuse-bridge.c:1237:fuse_err_cbk] 0-glusterfs-fuse: 22968: SETXATTR() /.gfid/826f2fc5-ddf1-4eb2-88c8-8ad10f4ead14 => -1 (File exists) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (In reply to Vijaykumar Koppad from comment #6) Could a couple of things be checked to eliminate variables? 1. Make sure the changelog records for the directory (under which all files are missing on the slave) are sane. 2. Anything in brick logs would help debugging. Please attach (or point) them too. In my setup, For the gfid say 'gfid1' on which "Operation not permitted" hits The file is there in master with 'gfid1' and on slave the file is present with no data with different gfid (completely random/corrupted gfid) which is not even there in master's backend changelog. So when rsync tries to sync data with original gfid, it is failing with 'Operation not Permitted' But no clue on how the entry of the file succeeded with completely different gfid. Upstream patch sent for review. http://review.gluster.org/#/c/7972/ Patch sent for review https://code.engineering.redhat.com/gerrit/#/c/26675/ These errors are not coming in the build glusterfs-3.6.0.20-1 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHEA-2014-1278.html |