Bug 1229422
| Summary: | server_lookup_cbk erros on LOOKUP only when quota is enabled | ||
|---|---|---|---|
| Product: | [Community] GlusterFS | Reporter: | Ryan Clough <ryan.clough> |
| Component: | quota | Assignee: | Raghavendra G <rgowdapp> |
| Status: | CLOSED EOL | QA Contact: | |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 3.7.1 | CC: | bkunal, bugs, ccalhoun, rgowdapp, ryan.clough, smohan |
| Target Milestone: | --- | Keywords: | Triaged |
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | rpc-ping-timeout | ||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-03-08 10:49:06 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Ryan Clough
2015-06-08 16:59:22 UTC
Now I am unable to run gluster commands: [root@hgluster01 ~]# gluster volume status Error : Request timed out [root@hgluster02 ~]# gluster volume status Error : Request timed out Also, disabling quota does not kill the initial quota crawl from enabling quota:
[root@hgluster01 ~]# ps aux | grep find
root 7961 1.1 0.0 112528 1228 ? S 08:57 0:48 /usr/bin/find . -exec /usr/bin/stat {} \ ;
root 28275 1.8 0.0 112528 1232 ? R 10:03 0:03 /usr/bin/find . -exec /usr/bin/setfattr -n glusterfs.quota-xattr-cleanup -v 1 {} \ ;
root 29955 0.0 0.0 103244 896 pts/2 S+ 10:06 0:00 grep find
Restarting glusterd on both nodes allows me to run gluster commands again. Could you attach the complete Gluster logs for both systems please? Upload limit is much to small to attach complete Gluster logs. I have made them available via my Google Drive: Logs from hgluster01: https://drive.google.com/file/d/0B_WRuYXQSKvrdUk4RVZ2enVqQ3c/view?usp=sharing Logs from hgluster02: https://drive.google.com/file/d/0B_WRuYXQSKvrMWxFZ2YxdWVvWjA/view?usp=sharing Were you able to download the logs? Still seeing these errors. Here is another example from the quota crawl log: [2015-07-08 22:40:06.802222] W [MSGID: 114031] [client-rpc-fops.c:2973:client3_3_lookup_cbk] 0-export_volume-client-1: remote operation failed: Permission denied. Path: /\ (00000000-0000-0000-0000-000000000000) [Permission denied] [2015-07-08 22:40:06.802304] W [fuse-bridge.c:484:fuse_entry_cbk] 0-glusterfs-fuse: 1966580: LOOKUP() /\ => -1 (Permission denied) [2015-07-08 23:24:53.044793] W [MSGID: 114031] [client-rpc-fops.c:2973:client3_3_lookup_cbk] 0-export_volume-client-1: remote operation failed: Permission denied. Path: /\ (00000000-0000-0000-0000-000000000000) [Permission denied] [2015-07-08 23:24:53.044868] W [fuse-resolve.c:67:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/\: failed to resolve (Permission denied) [2015-07-08 23:24:53.045194] W [fuse-bridge.c:484:fuse_entry_cbk] 0-glusterfs-fuse: 2583463: LOOKUP() /\ => -1 (Permission denied) [2015-07-08 23:24:53.048892] W [fuse-resolve.c:67:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/\: failed to resolve (Permission denied) [2015-07-08 23:24:53.049195] W [fuse-bridge.c:484:fuse_entry_cbk] 0-glusterfs-fuse: 2583465: LOOKUP() /\ => -1 (Permission denied) [2015-07-08 23:25:48.015659] W [fuse-resolve.c:67:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/\: failed to resolve (Permission denied) [2015-07-08 23:25:48.016025] W [fuse-bridge.c:484:fuse_entry_cbk] 0-glusterfs-fuse: 2589784: LOOKUP() /\ => -1 (Permission denied) [2015-07-08 23:25:48.019369] W [fuse-resolve.c:67:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/\: failed to resolve (Permission denied) [2015-07-08 23:25:48.019703] W [fuse-bridge.c:484:fuse_entry_cbk] 0-glusterfs-fuse: 2589786: LOOKUP() /\ => -1 (Permission denied) The message "W [MSGID: 114031] [client-rpc-fops.c:2973:client3_3_lookup_cbk] 0-export_volume-client-1: remote operation failed: Permission denied. Path: /\ (00000000-0000-0000-0000-000000000000) [Permission denied]" repeated 7 times between [2015-07-08 23:24:53.044793] and [2015-07-08 23:25:48.019687] [2015-07-09 00:33:53.186559] W [MSGID: 114031] [client-rpc-fops.c:2973:client3_3_lookup_cbk] 0-export_volume-client-0: remote operation failed: Permission denied. Path: /\ (00000000-0000-0000-0000-000000000000) [Permission denied] [2015-07-09 02:00:28.086666] W [MSGID: 114031] [client-rpc-fops.c:2973:client3_3_lookup_cbk] 0-export_volume-client-1: remote operation failed: Permission denied. Path: /\ (00000000-0000-0000-0000-000000000000) [Permission denied] [2015-07-09 02:00:28.095138] W [fuse-resolve.c:67:fuse_resolve_entry_cbk] 0-fuse: 00000000-0000-0000-0000-000000000001/\: failed to resolve (Permission denied) Below mentioned patches fixes the problem: http://review.gluster.org/#/c/13825/ http://review.gluster.org/#/c/13909/ http://review.gluster.org/#/c/13862/ Attached SF Case 01649316 as potentially related to this BZ. If someone can confirm or deny that it's the same issue, it would be appreciated. Please let me know if I can provide any additional information germane to making that assessment. This bug is getting closed because GlusteFS-3.7 has reached its end-of-life. Note: This bug is being closed using a script. No verification has been performed to check if it still exists on newer releases of GlusterFS. If this bug still exists in newer GlusterFS releases, please reopen this bug against the newer release. |