Bug 1248123
| Summary: | writes to glusterfs folder are not synced to other nodes unless they are explicitly read from gluster mount | ||
|---|---|---|---|
| Product: | [Community] GlusterFS | Reporter: | Ivo Petrov <cmptuomp3> |
| Component: | replicate | Assignee: | bugs <bugs> |
| Status: | CLOSED WORKSFORME | QA Contact: | |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 3.7.2 | CC: | atalur, bugs, cmptuomp3, gluster-bugs, kdhananj, pkarampu, ravishankar, rtalur |
| Target Milestone: | --- | Keywords: | Triaged |
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2015-09-24 01:55:33 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Ivo Petrov
2015-07-29 16:05:26 UTC
Forgot to add, my ssd_data.log is full with: [2015-07-26 14:05:43.676576] E [MSGID: 114031] [client-rpc-fops.c:251:client3_3_mknod_cbk] 0-ssd-gluster-data-client-2: remote operation failed: Permission denied. Path: (null) [Permission denied] [2015-07-26 14:05:43.841046] E [MSGID: 114031] [client-rpc-fops.c:251:client3_3_mknod_cbk] 0-ssd-gluster-data-client-1: remote operation failed: Permission denied. Path: (null) [Permission denied] [2015-07-26 14:05:43.841409] E [MSGID: 114031] [client-rpc-fops.c:251:client3_3_mknod_cbk] 0-ssd-gluster-data-client-2: remote operation failed: Permission denied. Path: (null) [Permission denied] [2015-07-26 14:05:44.251452] E [MSGID: 114031] [client-rpc-fops.c:251:client3_3_mknod_cbk] 0-ssd-gluster-data-client-1: remote operation failed: Permission denied. Path: (null) [Permission denied] [2015-07-26 14:05:44.251818] E [MSGID: 114031] [client-rpc-fops.c:251:client3_3_mknod_cbk] 0-ssd-gluster-data-client-2: remote operation failed: Permission denied. Path: (null) [Permission denied] [2015-07-26 14:05:44.643556] E [MSGID: 114031] [client-rpc-fops.c:321:client3_3_mkdir_cbk] 0-ssd-gluster-data-client-1: remote operation failed: Permission denied. Path: (null) [Permission denied] [2015-07-26 14:05:44.643940] E [MSGID: 114031] [client-rpc-fops.c:321:client3_3_mkdir_cbk] 0-ssd-gluster-data-client-2: remote operation failed: Permission denied. Path: (null) [Permission denied] [2015-07-26 14:05:44.816162] E [MSGID: 114031] [client-rpc-fops.c:251:client3_3_mknod_cbk] 0-ssd-gluster-data-client-1: remote operation failed: Permission denied. Path: (null) [Permission denied] that's on servers 10.0.1.1 and 10.0.1.2 10.0.1.3 has just: [2015-07-29 15:19:11.311369] W [fuse-bridge.c:1263:fuse_err_cbk] 0-glusterfs-fuse: 352139993: REMOVEXATTR() /somefile.. => -1 (No data available) selinux is disabled on all 3 servers Ivo, Could you share the brick logs from 10.0.1.1 and 10.0.1.2? Thanks, Krutika Hello Krutika, see my first comment here - that's all that's relevant in those logs. Ivo, Sorry I did not make myself clear. Could you share the logs at /var/log/glusterfs/bricks/*.log on 10.0.1.1 and 10.0.1.2? -Krutika Sorry - on 10.0.1.1 and 10.0.1.2 I have tons of [2015-08-04 13:10:28.801167] E [MSGID: 115057] [server-rpc-fops.c:535:server_mknod_cbk] 0-ssd-gluster-data-server: 193268: MKNOD /$FILE (04cc4296-7183-4efb-9bcc-7e6857eb2076/$FILE) ==> (Permission denied) [Permission denied] when healing (cat from glusterfs mount) log is full with: [2015-08-04 12:59:51.012863] I [dict.c:467:dict_get] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x186)[0x7f4c15b2cef6] (--> /usr/lib64/libglusterfs.so.0(dict_get+0x90)[0x7f4c15b25ce0] (--> /usr/lib64/glusterfs/3.7.2/xlator/storage/posix.so(posix_setxattr+0x1b9)[0x7f4c0823c439] (--> /usr/lib64/libglusterfs.so.0(default_setxattr+0x75)[0x7f4c15b313a5] (--> /usr/lib64/glusterfs/3.7.2/xlator/features/changetimerecorder.so(ctr_setxattr+0x186)[0x7f4c037cc696] ))))) 0-dict: !this || key=dht-get-iatt-in-xattr (In reply to Ivo Petrov from comment #5) > Sorry - on 10.0.1.1 and 10.0.1.2 I have tons of > > [2015-08-04 13:10:28.801167] E [MSGID: 115057] > [server-rpc-fops.c:535:server_mknod_cbk] 0-ssd-gluster-data-server: 193268: > MKNOD /$FILE (04cc4296-7183-4efb-9bcc-7e6857eb2076/$FILE) ==> (Permission > denied) [Permission denied] > > when healing (cat from glusterfs mount) log is full with: > > > [2015-08-04 12:59:51.012863] I [dict.c:467:dict_get] (--> > /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x186)[0x7f4c15b2cef6] (--> > /usr/lib64/libglusterfs.so.0(dict_get+0x90)[0x7f4c15b25ce0] (--> > /usr/lib64/glusterfs/3.7.2/xlator/storage/posix. > so(posix_setxattr+0x1b9)[0x7f4c0823c439] (--> > /usr/lib64/libglusterfs.so.0(default_setxattr+0x75)[0x7f4c15b313a5] (--> > /usr/lib64/glusterfs/3.7.2/xlator/features/changetimerecorder. > so(ctr_setxattr+0x186)[0x7f4c037cc696] ))))) 0-dict: !this || > key=dht-get-iatt-in-xattr This looks like an error that could *possibly* be originating from posix-acl translator. CC'ing Raghavendra Talur. Raghavendra, Would you be able to help us out here? Basically, entry creations on Ivo's 3-way replicated volume seem to be succeeding on only the first sub-volume of AFR and failing with EACCES (?) on the remaining two. (In reply to Ivo Petrov from comment #5) > when healing (cat from glusterfs mount) log is full with: > > > [2015-08-04 12:59:51.012863] I [dict.c:467:dict_get] (--> > /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x186)[0x7f4c15b2cef6] (--> > /usr/lib64/libglusterfs.so.0(dict_get+0x90)[0x7f4c15b25ce0] (--> > /usr/lib64/glusterfs/3.7.2/xlator/storage/posix. > so(posix_setxattr+0x1b9)[0x7f4c0823c439] (--> > /usr/lib64/libglusterfs.so.0(default_setxattr+0x75)[0x7f4c15b313a5] (--> > /usr/lib64/glusterfs/3.7.2/xlator/features/changetimerecorder. > so(ctr_setxattr+0x186)[0x7f4c037cc696] ))))) 0-dict: !this || > key=dht-get-iatt-in-xattr This is fixed in 3.7.3 version, you can look at this bug https://bugzilla.redhat.com/show_bug.cgi?id=1235923 (In reply to Ivo Petrov from comment #5) > Sorry - on 10.0.1.1 and 10.0.1.2 I have tons of > > [2015-08-04 13:10:28.801167] E [MSGID: 115057] > [server-rpc-fops.c:535:server_mknod_cbk] 0-ssd-gluster-data-server: 193268: > MKNOD /$FILE (04cc4296-7183-4efb-9bcc-7e6857eb2076/$FILE) ==> (Permission > denied) [Permission denied] For this original problem, my first question is about the user/process which is doing the write on gluster mount. Is it root or someone else? If someone else, does that user exist on the other two nodes? If non of the above the help us with determining whats wrong, we will have to check the brick logs at debug level. You can do that by executing "gluster volume set <VOLNAME> brick-log-level DEBUG". Ivo Petrov,
Are you still experiencing this bug? Could you provide the information asked by Raghavendra Talur in comment #8?
Pranith
Hello, I'm sorry, I've missed comment #8. I've tested this again and it seems to be working as expected now, the only difference is I had umount/mount the problematic mount on the node not from itself (local IP) but from another node, after that I checked consistency with a script I wrote - get all files from a "good node" get all files from "problematic node" diff the two lists and `cat to dev null` all files that are different on the prolematic node (cat is from glusterfs mount which forces them to be synced locally) After all that above, I can no longer reproduce the original issue. The mounts/setup is currently as in the original post and all is working as expected. Thanks Ivo Petrov. Closing the bug as per Comment #10 |