Bug 1325859

Summary: unable to use gluster - nfs-ganesha as nas for vmware
Product: [Community] GlusterFS Reporter: daniel de baerdemaeker <debaerd>
Component: ganesha-nfsAssignee: Jiffin <jthottan>
Status: CLOSED EOL QA Contact:
Severity: high Docs Contact:
Priority: low    
Version: 3.7.8CC: bugs
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-03-08 10:50:28 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description daniel de baerdemaeker 2016-04-11 11:20:53 UTC 
Description of problem:
When mounting a gluster volume as nfsv4 mount, using nfs-ganasha by vmware, i can not create virtual machines on it. If i try to create a folder it succeeds, but i am unable to remove the folder
I do not have the same problem when using native nfsv3 from gluster

Version-Release number of selected component (if applicable):
nfs-ganesha 2.3

How reproducible:
always

Steps to Reproduce:
1.export gluster volume by nfs-ganesha
2.mount it by vmware (from vcentre)
3.create folder
4.remove folder 


Actual results:
folder not deleted erro in log 

Expected results:
 folder deleted

Additional info:
firewall-cmd --list-ports
49152-49160/tcp 8652/tcp 111/udp 24007/tcp 4379/tcp 111/tcp 8651/tcp 8649/tcp 8649/udp 8652/udp 38465-38467/tcp 2049/tcp 8651/udp
[root@backup1 ganesha]# firewall-cmd --list-all
public (default, active)
  interfaces: em1 p6p1 p6p2 team0
  sources: 
  services: dhcpv6-client nfs ssh
  ports: 49152-49160/tcp 8652/tcp 111/udp 24007/tcp 4379/tcp 111/tcp 8651/tcp 8649/tcp 8649/udp 8652/udp 38465-38467/tcp 2049/tcp 8651/udp
  masquerade: no
  forward-ports: 
  icmp-blocks: 
  rich rules: 

/var/log/ganesha-gfapi.log


[2016-04-11 08:58:31.767059] I [MSGID: 101191] [event-epoll.c:663:event_dispatch_epoll_worker] 0-epoll: Exited thread with index 2
[2016-04-11 08:58:31.767091] I [MSGID: 101191] [event-epoll.c:663:event_dispatch_epoll_worker] 0-epoll: Exited thread with index 1
[2016-04-11 08:58:32.754927] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2016-04-11 08:58:32.769664] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2016-04-11 08:58:32.770124] I [MSGID: 104045] [glfs-master.c:95:notify] 0-gfapi: New graph 6261636b-7570-312e-6673-6f6666652e6f (0) coming up
[2016-04-11 08:58:32.770145] I [MSGID: 114020] [client.c:2106:notify] 0-gluvol0-client-0: parent translators are ready, attempting connect on transport
[2016-04-11 08:58:32.772418] I [MSGID: 114020] [client.c:2106:notify] 0-gluvol0-client-1: parent translators are ready, attempting connect on transport
[2016-04-11 08:58:32.772640] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-gluvol0-client-0: changing port to 49152 (from 0)
[2016-04-11 08:58:32.775004] I [rpc-clnt.c:1847:rpc_clnt_reconfig] 0-gluvol0-client-1: changing port to 49152 (from 0)
[2016-04-11 08:58:32.776991] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-gluvol0-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-04-11 08:58:32.777197] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-gluvol0-client-0: Connected to gluvol0-client-0, attached to remote volume '/mnt/data1/glustore'.
[2016-04-11 08:58:32.777220] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-gluvol0-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2016-04-11 08:58:32.777302] I [MSGID: 108005] [afr-common.c:3976:afr_notify] 0-gluvol0-replicate-0: Subvolume 'gluvol0-client-0' came back up; going online.
[2016-04-11 08:58:32.777355] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-gluvol0-client-0: Server lk version = 1
[2016-04-11 08:58:32.779141] I [MSGID: 114057] [client-handshake.c:1437:select_server_supported_programs] 0-gluvol0-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-04-11 08:58:32.779378] I [MSGID: 114046] [client-handshake.c:1213:client_setvolume_cbk] 0-gluvol0-client-1: Connected to gluvol0-client-1, attached to remote volume '/mnt/data1/glustore'.
[2016-04-11 08:58:32.779390] I [MSGID: 114047] [client-handshake.c:1224:client_setvolume_cbk] 0-gluvol0-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2016-04-11 08:58:32.784471] I [MSGID: 114035] [client-handshake.c:193:client_set_lk_version_cbk] 0-gluvol0-client-1: Server lk version = 1
[2016-04-11 08:58:32.785386] I [MSGID: 108031] [afr-common.c:1883:afr_local_discovery_cbk] 0-gluvol0-replicate-0: selecting local read_child gluvol0-client-0
[2016-04-11 08:58:32.785680] I [MSGID: 104041] [glfs-resolve.c:862:__glfs_active_subvol] 0-gluvol0: switched to graph 6261636b-7570-312e-6673-6f6666652e6f (0)
[2016-04-11 09:00:09.802304] I [MSGID: 109063] [dht-layout.c:702:dht_layout_normalize] 0-gluvol0-dht: Found anomalies in /.trashcan (gfid = 00000000-0000-0000-0000-000000000000). Holes=1 overlaps=0
[2016-04-11 09:00:09.803836] I [MSGID: 109036] [dht-common.c:8043:dht_log_new_layout_for_dir_selfheal] 0-gluvol0-dht: Setting layout of /.trashcan with [Subvol_name: gluvol0-replicate-0, Err: -1 , Start: 0 , Stop: 4294967295 , Hash: 1 ],
[2016-04-11 09:00:14.336260] I [MSGID: 109036] [dht-common.c:8043:dht_log_new_layout_for_dir_selfheal] 0-gluvol0-dht: Setting layout of /New Folder with [Subvol_name: gluvol0-replicate-0, Err: -1 , Start: 0 , Stop: 4294967295 , Hash: 1 ],


/var/log/ganesha.log

11/04/2016 10:58:32 : epoch 570b6490 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-23236[main] nfs_start :MAIN :EVENT :NFS EXIT: regular exit
11/04/2016 10:58:32 : epoch 570b6738 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-24829[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version /builddir/build/BUILD/nfs-ganesha-2.3.0/src, built at Oct 30 2015 12:03:29 on c1bk.rdu2.centos.org
11/04/2016 10:58:32 : epoch 570b6738 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-24830[main] nfs_set_param_from_conf :NFS STARTUP :EVENT :Configuration file successfully parsed
11/04/2016 10:58:32 : epoch 570b6738 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-24830[main] init_server_pkgs :NFS STARTUP :EVENT :Initializing ID Mapper.
11/04/2016 10:58:32 : epoch 570b6738 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-24830[main] init_server_pkgs :NFS STARTUP :EVENT :ID Mapper successfully initialized.
11/04/2016 10:58:32 : epoch 570b6738 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-24830[main] glusterfs_create_export :FSAL :EVENT :Volume gluvol0 exported at : '/'
11/04/2016 10:58:33 : epoch 570b6738 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-24830[main] lower_my_caps :NFS STARTUP :EVENT :CAP_SYS_RESOURCE was successfully removed for proper quota management in FSAL
11/04/2016 10:58:33 : epoch 570b6738 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-24830[main] lower_my_caps :NFS STARTUP :EVENT :currenty set capabilities are: = cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap+ep
11/04/2016 10:58:33 : epoch 570b6738 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-24830[main] nfs_Init_svc :DISP :CRIT :Cannot acquire credentials for principal nfs
11/04/2016 10:58:33 : epoch 570b6738 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-24830[main] nfs_Init_admin_thread :NFS CB :EVENT :Admin thread initialized
11/04/2016 10:58:33 : epoch 570b6738 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-24830[main] nfs4_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 60
11/04/2016 10:58:33 : epoch 570b6738 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-24830[main] nfs_rpc_cb_init_ccache :NFS STARTUP :EVENT :Callback creds directory (/var/run/ganesha) already exists
11/04/2016 10:58:33 : epoch 570b6738 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-24830[main] nfs_rpc_cb_init_ccache :NFS STARTUP :WARN :gssd_refresh_krb5_machine_credential failed (-1765328160:0)
11/04/2016 10:58:33 : epoch 570b6738 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-24830[main] nfs_Start_threads :THREAD :EVENT :Starting delayed executor.
11/04/2016 10:58:33 : epoch 570b6738 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-24830[main] nfs_Start_threads :THREAD :EVENT :9P/TCP dispatcher thread was started successfully
11/04/2016 10:58:33 : epoch 570b6738 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-24830[main] nfs_Start_threads :THREAD :EVENT :gsh_dbusthread was started successfully
11/04/2016 10:58:33 : epoch 570b6738 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-24830[_9p_disp] _9p_dispatcher_thread :9P DISP :EVENT :9P dispatcher started
11/04/2016 10:58:33 : epoch 570b6738 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-24830[main] nfs_Start_threads :THREAD :EVENT :admin thread was started successfully
11/04/2016 10:58:33 : epoch 570b6738 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-24830[main] nfs_Start_threads :THREAD :EVENT :reaper thread was started successfully
11/04/2016 10:58:33 : epoch 570b6738 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-24830[main] nfs_Start_threads :THREAD :EVENT :General fridge was started successfully
11/04/2016 10:58:33 : epoch 570b6738 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-24830[reaper] nfs_in_grace :STATE :EVENT :NFS Server Now IN GRACE
11/04/2016 10:58:33 : epoch 570b6738 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-24830[main] nfs_start :NFS STARTUP :EVENT :-------------------------------------------------
11/04/2016 10:58:33 : epoch 570b6738 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-24830[main] nfs_start :NFS STARTUP :EVENT :             NFS SERVER INITIALIZED
11/04/2016 10:58:33 : epoch 570b6738 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-24830[main] nfs_start :NFS STARTUP :EVENT :-------------------------------------------------
11/04/2016 10:59:33 : epoch 570b6738 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-24830[reaper] nfs_in_grace :STATE :EVENT :NFS Server Now NOT IN GRACE
Comment 1 daniel de baerdemaeker 2016-04-14 10:12:26 UTC 
the split brain came from using a bad hostname in the fsal, as we use a different vlan for replication

after playing with the command line on the vmware hosts, i get the folowing

[root@VS2DRC:/vmfs/volumes/b79958f6-f939a861-0000-000000000000/New Folder] ls -l
a
total 16
drwxr-xr-x    4 root     root          4096 Apr 14 09:15 .
drwxr-xr-x    7 root     root          4096 Apr 14 08:42 ..
drwxr-xr-x    2 root     root          4096 Apr 14 09:15 abc
drwxr-xr-x    2 root     root          4096 Apr 14 09:05 ggggg


[root@VS2DRC:/vmfs/volumes/b79958f6-f939a861-0000-000000000000/New Folder] rmdir
 ggggg/
rmdir: 'ggggg/': Not a directory

if i change to kernel-nfs, i can remove these folders. But the performance is bad
Comment 2 daniel de baerdemaeker 2016-04-15 14:22:45 UTC 
after putting on debuging i get this when removing a directory (from commandline in vmware)

2016-04-15 04:15:16 PM : epoch 5710f279 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-121004[::ffff:192.168.60.204] [work-13] /builddir/build/BUILD/nfs-ganesha-2.3.1/src/Protocols/NFS/nfs4_Compound.c :nfs4_Compound :NFS4 :DEBUG :COMPOUND: There are 7 operations
2016-04-15 04:15:16 PM : epoch 5710f279 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-121004[::ffff:192.168.60.204] [work-13] /builddir/build/BUILD/nfs-ganesha-2.3.1/src/Protocols/NFS/nfs4_Compound.c :nfs4_Compound :NFS4 :DEBUG :Request 0: opcode 53 is OP_SEQUENCE
2016-04-15 04:15:16 PM : epoch 5710f279 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-121004[::ffff:192.168.60.204] [work-13] /builddir/build/BUILD/nfs-ganesha-2.3.1/src/Protocols/NFS/nfs4_op_sequence.c :nfs4_op_sequence :SESSIONS :DEBUG :SEQUENCE session=0x7fe8c2046100
2016-04-15 04:15:16 PM : epoch 5710f279 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-121004[::ffff:192.168.60.204] [work-13] /builddir/build/BUILD/nfs-ganesha-2.3.1/src/Protocols/NFS/nfs4_Compound.c :nfs4_Compound :NFS4 :DEBUG :Request 1: opcode 22 is OP_PUTFH
2016-04-15 04:15:16 PM : epoch 5710f279 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-121004[::ffff:192.168.60.204] [work-13] /builddir/build/BUILD/nfs-ganesha-2.3.1/src/support/export_mgr.c :get_gsh_export :HT CACHE :DEBUG :export_mgr cache hit slot 100
2016-04-15 04:15:16 PM : epoch 5710f279 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-121004[::ffff:192.168.60.204] [work-13] /builddir/build/BUILD/nfs-ganesha-2.3.1/src/support/nfs_creds.c :nfs4_export_check_access :NFS4 :M_DBG :NFS4: MID DEBUG: nfs4_export_check_access about to call export_check_access
2016-04-15 04:15:16 PM : epoch 5710f279 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-121004[::ffff:192.168.60.204] [work-13] /builddir/build/BUILD/nfs-ganesha-2.3.1/src/support/exports.c :export_check_access :EXPORT :M_DBG :Check for address 192.168.60.204 for export id 100 fullpath /gluvol0
2016-04-15 04:15:16 PM : epoch 5710f279 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-121004[::ffff:192.168.60.204] [work-13] /builddir/build/BUILD/nfs-ganesha-2.3.1/src/support/exports.c :client_match :EXPORT :M_DBG :Match 0x7fe8ed410800, type = NETWORK_CLIENT, options 0x77030F1
2016-04-15 04:15:16 PM : epoch 5710f279 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-121004[::ffff:192.168.60.204] [work-13] /builddir/build/BUILD/nfs-ganesha-2.3.1/src/support/exports.c :LogClientListEntry :EXPORT :M_DBG :  0x7fe8ed410800 NETWORK_CLIENT: 0.60.168.192 (no_root_squash, RWrw,    ,               ,               ,         ,                ,                )
2016-04-15 04:15:16 PM : epoch 5710f279 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-121004[::ffff:192.168.60.204] [work-13] /builddir/build/BUILD/nfs-ganesha-2.3.1/src/support/exports.c :export_check_access :EXPORT :M_DBG :CLIENT          (no_root_squash, RWrw,    ,               ,               ,         ,                ,                )
2016-04-15 04:15:16 PM : epoch 5710f279 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-121004[::ffff:192.168.60.204] [work-13] /builddir/build/BUILD/nfs-ganesha-2.3.1/src/support/exports.c :export_check_access :EXPORT :M_DBG :EXPORT          (no_root_squash, RWrw, -4-, ---, TCP, ----,               ,         ,                ,                , sys)
2016-04-15 04:15:16 PM : epoch 5710f279 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-121004[::ffff:192.168.60.204] [work-13] /builddir/build/BUILD/nfs-ganesha-2.3.1/src/support/exports.c :export_check_access :EXPORT :M_DBG :EXPORT_DEFAULTS (              ,     ,    ,               ,               ,         ,                ,                )
2016-04-15 04:15:16 PM : epoch 5710f279 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-121004[::ffff:192.168.60.204] [work-13] /builddir/build/BUILD/nfs-ganesha-2.3.1/src/support/exports.c :export_check_access :EXPORT :M_DBG :default options (root_squash   , ----, 34-, UDP, TCP, ----, No Manage_Gids, -- Deleg, anon_uid=    -2, anon_gid=    -2, none, sys)
2016-04-15 04:15:16 PM : epoch 5710f279 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-121004[::ffff:192.168.60.204] [work-13] /builddir/build/BUILD/nfs-ganesha-2.3.1/src/support/exports.c :export_check_access :EXPORT :M_DBG :Final options   (no_root_squash, RWrw, -4-, ---, TCP, ----, No Manage_Gids, -- Deleg, anon_uid=    -2, anon_gid=    -2, sys)
2016-04-15 04:15:16 PM : epoch 5710f279 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-121004[::ffff:192.168.60.204] [work-13] /builddir/build/BUILD/nfs-ganesha-2.3.1/src/support/nfs_creds.c :nfs_req_creds :DISP :M_DBG :DISP: MID DEBUG: AUTH_SYS creds mapped to uid=0, gid=0, glen=1
2016-04-15 04:15:16 PM : epoch 5710f279 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-121004[::ffff:192.168.60.204] [work-13] /builddir/build/BUILD/nfs-ganesha-2.3.1/src/include/cache_inode_hash.h :cih_get_by_key_latched :HT CACHE :DEBUG :cih cache hit slot 7303
2016-04-15 04:15:16 PM : epoch 5710f279 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-121004[::ffff:192.168.60.204] [work-13] /builddir/build/BUILD/nfs-ganesha-2.3.1/src/Protocols/NFS/nfs4_Compound.c :nfs4_Compound :NFS4 :DEBUG :Request 2: opcode 15 is OP_LOOKUP
2016-04-15 04:15:16 PM : epoch 5710f279 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-121004[::ffff:192.168.60.204] [work-13] /builddir/build/BUILD/nfs-ganesha-2.3.1/src/Protocols/NFS/nfs4_Compound.c :nfs4_Compound :NFS4 :M_DBG :NFS4: MID DEBUG: Check export perms export = 000000f0 req = 00000040
2016-04-15 04:15:16 PM : epoch 5710f279 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-121004[::ffff:192.168.60.204] [work-13] /builddir/build/BUILD/nfs-ganesha-2.3.1/src/Protocols/NFS/nfs4_op_lookup.c :nfs4_op_lookup :NFS4 :DEBUG :name=abc
2016-04-15 04:15:16 PM : epoch 5710f279 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-121004[::ffff:192.168.60.204] [work-13] /builddir/build/BUILD/nfs-ganesha-2.3.1/src/cache_inode/cache_inode_access.c :cache_inode_access_sw :INODE :DEBUG :INODE: DEBUG: access_type=0X81000020
2016-04-15 04:15:16 PM : epoch 5710f279 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-121004[::ffff:192.168.60.204] [work-13] /builddir/build/BUILD/nfs-ganesha-2.3.1/src/include/cache_inode_hash.h :cih_get_by_key_latched :HT CACHE :DEBUG :cih cache hit slot 17079
2016-04-15 04:15:16 PM : epoch 5710f279 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-121004[::ffff:192.168.60.204] [work-13] /builddir/build/BUILD/nfs-ganesha-2.3.1/src/Protocols/NFS/nfs4_Compound.c :nfs4_Compound :NFS4 :DEBUG :Request 3: opcode 37 is OP_VERIFY
2016-04-15 04:15:16 PM : epoch 5710f279 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-121004[::ffff:192.168.60.204] [work-13] /builddir/build/BUILD/nfs-ganesha-2.3.1/src/Protocols/NFS/nfs4_Compound.c :nfs4_Compound :NFS4 :M_DBG :NFS4: MID DEBUG: Check export perms export = 000000f0 req = 00000040
2016-04-15 04:15:16 PM : epoch 5710f279 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-121004[::ffff:192.168.60.204] [work-13] /builddir/build/BUILD/nfs-ganesha-2.3.1/src/Protocols/NFS/nfs_proto_tools.c :cache_entry_To_Fattr :NFS4 ACL :DEBUG :No permission check for ACL for entry 0x7fe8c2060e80
2016-04-15 04:15:16 PM : epoch 5710f279 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-121004[::ffff:192.168.60.204] [work-13] /builddir/build/BUILD/nfs-ganesha-2.3.1/src/Protocols/NFS/nfs4_Compound.c :nfs4_Compound :NFS4 :DEBUG :Status of OP_VERIFY in position 3 = NFS4ERR_NOT_SAME
2016-04-15 04:15:16 PM : epoch 5710f279 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-121004[::ffff:192.168.60.204] [work-13] /builddir/build/BUILD/nfs-ganesha-2.3.1/src/Protocols/NFS/nfs4_Compound.c :nfs4_Compound :NFS4 :DEBUG :End status = NFS4ERR_NOT_SAME lastindex = 3

when doing this from a linux machine i get

2016-04-15 04:21:01 PM : epoch 5710f279 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-121004[::ffff:192.168.60.153] [work-2] /builddir/build/BUILD/nfs-ganesha-2.3.1/src/support/exports.c :export_check_access :EXPORT :M_DBG :Check for address 192.168.60.153 for export id 100 fullpath /gluvol0
2016-04-15 04:21:01 PM : epoch 5710f279 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-121004[::ffff:192.168.60.153] [work-2] /builddir/build/BUILD/nfs-ganesha-2.3.1/src/support/exports.c :client_match :EXPORT :M_DBG :Match 0x7fe8ed410800, type = NETWORK_CLIENT, options 0x77030F1
2016-04-15 04:21:01 PM : epoch 5710f279 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-121004[::ffff:192.168.60.153] [work-2] /builddir/build/BUILD/nfs-ganesha-2.3.1/src/support/exports.c :LogClientListEntry :EXPORT :M_DBG :  0x7fe8ed410800 NETWORK_CLIENT: 0.60.168.192 (no_root_squash, RWrw,    ,               ,               ,         ,                ,                )
2016-04-15 04:21:01 PM : epoch 5710f279 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-121004[::ffff:192.168.60.153] [work-2] /builddir/build/BUILD/nfs-ganesha-2.3.1/src/support/exports.c :export_check_access :EXPORT :M_DBG :CLIENT          (no_root_squash, RWrw,    ,               ,               ,         ,                ,                )
2016-04-15 04:21:01 PM : epoch 5710f279 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-121004[::ffff:192.168.60.153] [work-2] /builddir/build/BUILD/nfs-ganesha-2.3.1/src/support/exports.c :export_check_access :EXPORT :M_DBG :EXPORT          (no_root_squash, RWrw, -4-, ---, TCP, ----,               ,         ,                ,                , sys)
2016-04-15 04:21:01 PM : epoch 5710f279 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-121004[::ffff:192.168.60.153] [work-2] /builddir/build/BUILD/nfs-ganesha-2.3.1/src/support/exports.c :export_check_access :EXPORT :M_DBG :EXPORT_DEFAULTS (              ,     ,    ,               ,               ,         ,                ,                )
2016-04-15 04:21:01 PM : epoch 5710f279 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-121004[::ffff:192.168.60.153] [work-2] /builddir/build/BUILD/nfs-ganesha-2.3.1/src/support/exports.c :export_check_access :EXPORT :M_DBG :default options (root_squash   , ----, 34-, UDP, TCP, ----, No Manage_Gids, -- Deleg, anon_uid=    -2, anon_gid=    -2, none, sys)
2016-04-15 04:21:01 PM : epoch 5710f279 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-121004[::ffff:192.168.60.153] [work-2] /builddir/build/BUILD/nfs-ganesha-2.3.1/src/support/exports.c :export_check_access :EXPORT :M_DBG :Final options   (no_root_squash, RWrw, -4-, ---, TCP, ----, No Manage_Gids, -- Deleg, anon_uid=    -2, anon_gid=    -2, sys)
2016-04-15 04:21:01 PM : epoch 5710f279 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-121004[::ffff:192.168.60.153] [work-2] /builddir/build/BUILD/nfs-ganesha-2.3.1/src/support/nfs_creds.c :nfs_req_creds :DISP :M_DBG :DISP: MID DEBUG: AUTH_SYS creds mapped to uid=0, gid=0, glen=2
2016-04-15 04:21:01 PM : epoch 5710f279 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-121004[::ffff:192.168.60.153] [work-2] /builddir/build/BUILD/nfs-ganesha-2.3.1/src/include/cache_inode_hash.h :cih_get_by_key_latched :HT CACHE :DEBUG :cih cache hit slot 7303
2016-04-15 04:21:01 PM : epoch 5710f279 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-121004[::ffff:192.168.60.153] [work-2] /builddir/build/BUILD/nfs-ganesha-2.3.1/src/Protocols/NFS/nfs4_Compound.c :nfs4_Compound :NFS4 :DEBUG :Request 1: opcode 28 is OP_REMOVE
2016-04-15 04:21:01 PM : epoch 5710f279 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-121004[::ffff:192.168.60.153] [work-2] /builddir/build/BUILD/nfs-ganesha-2.3.1/src/Protocols/NFS/nfs4_Compound.c :nfs4_Compound :NFS4 :M_DBG :NFS4: MID DEBUG: Check export perms export = 000000f0 req = 00000080
2016-04-15 04:21:01 PM : epoch 5710f279 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-121004[::ffff:192.168.60.153] [work-2] /builddir/build/BUILD/nfs-ganesha-2.3.1/src/include/cache_inode_hash.h :cih_get_by_key_latched :HT CACHE :DEBUG :cih cache hit slot 17079
2016-04-15 04:21:01 PM : epoch 5710f279 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-121004[::ffff:192.168.60.153] [work-2] /builddir/build/BUILD/nfs-ganesha-2.3.1/src/cache_inode/cache_inode_remove.c :cache_inode_remove :INODE :DEBUG :abc
2016-04-15 04:21:01 PM : epoch 5710f279 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-121004[::ffff:192.168.60.153] [work-2] /builddir/build/BUILD/nfs-ganesha-2.3.1/src/cache_inode/cache_inode_remove.c :cache_inode_remove :INODE :DEBUG :cache_inode_remove_cached_dirent abc status CACHE_INODE_SUCCESS
2016-04-15 04:21:01 PM : epoch 5710f279 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-121004[::ffff:192.168.60.153] [work-2] /builddir/build/BUILD/nfs-ganesha-2.3.1/src/cache_inode/cache_inode_kill_entry.c :cache_inode_kill_entry :INODE :DEBUG :Using cache_inode_kill_entry for entry 0x7fe8c2060e80
2016-04-15 04:21:01 PM : epoch 5710f279 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-121004[::ffff:192.168.60.153] [work-2] /builddir/build/BUILD/nfs-ganesha-2.3.1/src/include/cache_inode.h :cache_inode_refresh_attrs :INODE :DEBUG :Failed on entry 0x7fe8c2060e80 CACHE_INODE_ESTALE
2016-04-15 04:21:01 PM : epoch 5710f279 : backup1.fsoffe.onerva.fgov.be : ganesha.nfsd-121004[::ffff:192.168.60.153] [work-2] /builddir/build/BUILD/nfs-ganesha-2.3.1/src/cache_inode/cache_inode_lru.c :cache_inode_lru_unref :INODE :DEBUG :LRU_ENTRY_CLEANUP of entry 0x7fe8c2060e80
Comment 3 Kaushal 2017-03-08 10:50:28 UTC 
This bug is getting closed because GlusteFS-3.7 has reached its end-of-life.

Note: This bug is being closed using a script. No verification has been performed to check if it still exists on newer releases of GlusterFS.
If this bug still exists in newer GlusterFS releases, please reopen this bug against the newer release.