Bug 1386236
| Summary: | Contents of subdir v3 mount can't be listed after failover. | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Gluster Storage | Reporter: | Shashank Raj <sraj> |
| Component: | nfs-ganesha | Assignee: | Jiffin <jthottan> |
| Status: | CLOSED ERRATA | QA Contact: | Arthy Loganathan <aloganat> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | rhgs-3.2 | CC: | jthottan, kkeithle, mzywusko, ndevos, rcyriac, rhinduja, rhs-bugs, sbhaloth, skoduri, storage-qa-internal |
| Target Milestone: | --- | ||
| Target Release: | RHGS 3.2.0 | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | nfs-ganesha-2.4.1-1 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-03-23 06:24:28 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1351528 | ||
The issue is seen only in v3 mount.(I guess, it should be reproduced in FSAL_VFS as well, I didn't try it) Steps to reproduce : export same gluster volume via two different servers say server1 and server2 mount the volume using server1 and create a subdir inside it now mount the volume on server2 using that subdir and then perform ls on the mount point which throws the same error as mentioned above Initial RCA : In nfs3_readdirplus() performs a lookup on parent directory which returns invalid error for parent. In mdc_lookup, ganehsa is trying to read parent entry from cache(but it is not cached before) which end up in this failure
nfs3_readdirplus->fsal_lookupp->mdcache_lookup->mdc_lookup->mdcache_locate_keyed
(here value of name is passed as "..")
At mdcache_locate_keyed when try to find cache entry using key results in EINVAL
(valu of key->kv is NULL)
The patch got merged upstream https://review.gerrithub.io/#/c/299022/ and available in latest nfs-ganesha rpms Executed the tests with both nfs v3 and v4 mount combinations and is passing now. Verified the fix in build, nfs-ganesha-2.4.1-1.el7rhgs.x86_64 glusterfs-ganesha-3.8.4-5.el7rhgs.x86_64 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHEA-2017-0493.html |
Description of problem: Contents of subdir v3 mount can't be listed after failover. Version-Release number of selected component (if applicable): [root@dhcp43-110 ~]# rpm -qa|grep glusterfs glusterfs-client-xlators-3.8.4-2.el7rhgs.x86_64 glusterfs-fuse-3.8.4-2.el7rhgs.x86_64 glusterfs-ganesha-3.8.4-2.el7rhgs.x86_64 glusterfs-cli-3.8.4-2.el7rhgs.x86_64 glusterfs-3.8.4-2.el7rhgs.x86_64 glusterfs-server-3.8.4-2.el7rhgs.x86_64 glusterfs-events-3.8.4-2.el7rhgs.x86_64 glusterfs-libs-3.8.4-2.el7rhgs.x86_64 glusterfs-geo-replication-3.8.4-2.el7rhgs.x86_64 glusterfs-debuginfo-3.8.4-1.el7rhgs.x86_64 glusterfs-api-3.8.4-2.el7rhgs.x86_64 [root@dhcp43-110 ~]# rpm -qa|grep ganesha nfs-ganesha-debuginfo-2.4.0-2.el7rhgs.x86_64 nfs-ganesha-2.4.0-2.el7rhgs.x86_64 glusterfs-ganesha-3.8.4-2.el7rhgs.x86_64 nfs-ganesha-gluster-2.4.0-2.el7rhgs.x86_64 How reproducible: Always Steps to Reproduce: 1.Create a ganesha cluster, create a volume and disable client-io thread on the volume: [root@dhcp43-92 ~]# gluster vol get mdcache all | grep client-io performance.client-io-threads off 2.Enable ganesha on the volume and do a subdir mount using v3 on the client. [root@Client2 ~]# mount -t nfs -o vers=4 10.70.40.192:/mdcache /mnt/nfs1 [root@Client2 ~]# cd /mnt/nfs1 [root@Client2 nfs1]# ls [root@Client2 nfs1]# mkdir subdir [root@Client2 nfs1]# ls subdir [root@Client2 nfs1]# cd [root@Client2 ~]# df Filesystem 1K-blocks Used Available Use% Mounted on 10.70.40.192:/mdcache 125107200 202752 124904448 1% /mnt/nfs1 [root@Client2 ~]# umount /mnt/nfs1 [root@Client2 ~]# [root@Client2 ~]# mount -t nfs -o vers=4 10.70.40.192:/mdcache/subdir /mnt/nfs1 [root@Client2 ~]# df Filesystem 1K-blocks Used Available Use% Mounted on 10.70.40.192:/mdcache/subdir 125107200 202752 124904448 1% /mnt/nfs1 3. Start IO on subdir mount: [root@Client2 ~]# for i in {1..30}; do mkdir /mnt/nfs1/d$i; for j in {1..50}; do mkdir /mnt/nfs1/d$i/e$j; for k in {1..50}; do touch /mnt/nfs1/d$i/e$j/f$k; done done done 4. Keep performing ls on the mount point: [root@Client2 nfs1]# ls d1 [root@Client2 nfs1]# ls d1 [root@Client2 nfs1]# ls d1 d2 [root@Client2 nfs1]# ls d1 d2 [root@Client2 nfs1]# ls d1 d2 d3 [root@Client2 nfs1]# ls d1 d2 d3 [root@Client2 nfs1]# ls d1 d2 d3 d4 [root@Client2 nfs1]# ls d1 d2 d3 d4 d5 Observe that data gets listed without any issues. 5. Stop nfs-ganesha on the mounted node and make sure failover happens: [root@dhcp43-92 ~]# service nfs-ganesha stop Redirecting to /bin/systemctl stop nfs-ganesha.service [root@dhcp43-92 ~]# service nfs-ganesha status Redirecting to /bin/systemctl status nfs-ganesha.service ● nfs-ganesha.service - NFS-Ganesha file server Loaded: loaded (/usr/lib/systemd/system/nfs-ganesha.service; disabled; vendor preset: disabled) Active: inactive (dead) since Tue 2016-10-18 08:03:08 EDT; 10s ago Docs: http://github.com/nfs-ganesha/nfs-ganesha/wiki Process: 14195 ExecStop=/bin/dbus-send --system --dest=org.ganesha.nfsd --type=method_call /org/ganesha/nfsd/admin org.ganesha.nfsd.admin.shutdown (code=exited, status=0/SUCCESS) Process: 11269 ExecStartPost=/bin/bash -c prlimit --pid $MAINPID --nofile=$NOFILE:$NOFILE (code=exited, status=0/SUCCESS) Process: 11267 ExecStart=/bin/bash -c ${NUMACTL} ${NUMAOPTS} /usr/bin/ganesha.nfsd ${OPTIONS} ${EPOCH} (code=exited, status=0/SUCCESS) Main PID: 11268 (code=exited, status=0/SUCCESS) Oct 18 07:57:28 dhcp43-92.lab.eng.blr.redhat.com systemd[1]: Starting NFS-Ganesh... Oct 18 07:57:28 dhcp43-92.lab.eng.blr.redhat.com systemd[1]: Started NFS-Ganesha... Oct 18 08:02:57 dhcp43-92.lab.eng.blr.redhat.com systemd[1]: Stopping NFS-Ganesh... Oct 18 08:03:08 dhcp43-92.lab.eng.blr.redhat.com systemd[1]: Stopped NFS-Ganesha... Hint: Some lines were ellipsized, use -l to show in full. [root@dhcp43-92 ~]# pcs status Cluster name: G1476787778.16 Last updated: Tue Oct 18 08:03:41 2016 Last change: Tue Oct 18 08:03:17 2016 by root via crm_attribute on dhcp43-92.lab.eng.blr.redhat.com Stack: corosync Current DC: dhcp43-92.lab.eng.blr.redhat.com (version 1.1.13-10.el7_2.2-44eb2dd) - partition with quorum 4 nodes and 16 resources configured Online: [ dhcp42-170.lab.eng.blr.redhat.com dhcp42-183.lab.eng.blr.redhat.com dhcp43-145.lab.eng.blr.redhat.com dhcp43-92.lab.eng.blr.redhat.com ] Full list of resources: Clone Set: nfs_setup-clone [nfs_setup] Started: [ dhcp42-170.lab.eng.blr.redhat.com dhcp42-183.lab.eng.blr.redhat.com dhcp43-145.lab.eng.blr.redhat.com dhcp43-92.lab.eng.blr.redhat.com ] Clone Set: nfs-mon-clone [nfs-mon] Started: [ dhcp42-170.lab.eng.blr.redhat.com dhcp42-183.lab.eng.blr.redhat.com dhcp43-145.lab.eng.blr.redhat.com dhcp43-92.lab.eng.blr.redhat.com ] Clone Set: nfs-grace-clone [nfs-grace] Started: [ dhcp42-170.lab.eng.blr.redhat.com dhcp42-183.lab.eng.blr.redhat.com dhcp43-145.lab.eng.blr.redhat.com ] Stopped: [ dhcp43-92.lab.eng.blr.redhat.com ] dhcp43-92.lab.eng.blr.redhat.com-cluster_ip-1 (ocf::heartbeat:IPaddr): Started dhcp42-183.lab.eng.blr.redhat.com dhcp42-170.lab.eng.blr.redhat.com-cluster_ip-1 (ocf::heartbeat:IPaddr): Started dhcp42-170.lab.eng.blr.redhat.com dhcp43-145.lab.eng.blr.redhat.com-cluster_ip-1 (ocf::heartbeat:IPaddr): Started dhcp43-145.lab.eng.blr.redhat.com dhcp42-183.lab.eng.blr.redhat.com-cluster_ip-1 (ocf::heartbeat:IPaddr): Started dhcp42-183.lab.eng.blr.redhat.com PCSD Status: dhcp43-92.lab.eng.blr.redhat.com: Online dhcp42-170.lab.eng.blr.redhat.com: Online dhcp43-145.lab.eng.blr.redhat.com: Online dhcp42-183.lab.eng.blr.redhat.com: Online Daemon Status: corosync: active/disabled pacemaker: active/disabled pcsd: active/disabled [root@dhcp43-92 ~]# 6. Perform ls again on the mount point and observe that it gives "ls: reading directory .: Invalid argument": [root@Client2 nfs1]# ls ls: reading directory .: Invalid argument [root@Client2 nfs1]# ls ls: reading directory .: Invalid argument [root@Client2 nfs1]# ls ls: reading directory .: Invalid argument [root@Client2 nfs1]# ls ls: reading directory .: Invalid argument [root@Client2 nfs1]# ls ls: reading directory .: Invalid argument [root@Client2 nfs1]# ls ls: reading directory .: Invalid argument 7. However cd to a dir works fine: [root@Client2 nfs1]# cd d1 [root@Client2 d1]# ls e1 e11 e13 e15 e17 e19 e20 e22 e24 e26 e28 e3 e31 e33 e35 e37 e39 e40 e42 e44 e46 e48 e5 e6 e8 e10 e12 e14 e16 e18 e2 e21 e23 e25 e27 e29 e30 e32 e34 e36 e38 e4 e41 e43 e45 e47 e49 e50 e7 e9 Actual results: Contents of subdir v3 mount can't be listed after failover Expected results: ls should list the content even after failover. Additional info: