Bug 1421759
| Summary: | Gluster NFS server crashing in __mnt3svc_umountall | |||
|---|---|---|---|---|
| Product: | [Community] GlusterFS | Reporter: | Raghavendra Bhat <rabhat> | |
| Component: | nfs | Assignee: | Soumya Koduri <skoduri> | |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | ||
| Severity: | urgent | Docs Contact: | ||
| Priority: | urgent | |||
| Version: | mainline | CC: | acavalla, akaiser, amukherj, aperotti, bkunal, bugs, ctowsley, dnunes, fahmed, fgaspar, ndevos, olim, omasek, rabhat, rhs-bugs, rnalakka, skoduri, storage-qa-internal | |
| Target Milestone: | --- | Keywords: | Triaged | |
| Target Release: | --- | |||
| Hardware: | All | |||
| OS: | All | |||
| Whiteboard: | ||||
| Fixed In Version: | glusterfs-3.11.0 | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | 1315544 | |||
| : | 1422391 (view as bug list) | Environment: | ||
| Last Closed: | 2017-05-30 18:43:01 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1315544, 1422391, 1422392, 1422394, 1422404 | |||
|
Description
Raghavendra Bhat
2017-02-13 15:38:27 UTC
Possible fix(es) are in attachment #1160478 [details] from bug 1315544. Raghavendra, will you be verifying those changes and post them for review? REVIEW: https://review.gluster.org/16611 (gNFS: Keep the mountdict as long as the service is active) posted (#2) for review on master by soumya k (skoduri) COMMIT: https://review.gluster.org/16611 committed in master by Niels de Vos (ndevos) ------ commit a88ae92de190af0956013780939ba6bdfd509ff8 Author: Soumya Koduri <skoduri> Date: Mon Feb 13 16:52:07 2017 +0530 gNFS: Keep the mountdict as long as the service is active We initialize and take ref once on mountdict during NFS/MNT3 server initialization but seem to be unref'in it for every UMNTALL request. This can lead to crash when there are multiple UMNTALL requests with >=1 active mount entry(/ies) in the mountlist. Since we take the ref only once, we should keep the mountdict through out the life of the process and dereference it only during unitialization of mnt3 service. Change-Id: I3238a8df09b8972e56dd93fee426d866d40d9959 BUG: 1421759 Signed-off-by: Soumya Koduri <skoduri> Reviewed-on: https://review.gluster.org/16611 Smoke: Gluster Build System <jenkins.org> Reviewed-by: Raghavendra Bhat <raghavendra> NetBSD-regression: NetBSD Build System <jenkins.org> CentOS-regression: Gluster Build System <jenkins.org> Reviewed-by: jiffin tony Thottan <jthottan> Reviewed-by: Niels de Vos <ndevos> You have a good theory on how this use-after-free can occur. You should be able to reproduce this with nfsshell from https://github.com/NetDirect/nfsshell as it can do UMNTALL too. Using this program it should be possible to trigger the problem. An example for running the shell is like: $ ./nfsshell nfs> host storage.lan.nixpanic.net Open storage.lan.example.net (172.31.96.2) TCP nfs> mount /repos Mount `/repos', TCP, transfer size 1048576 bytes. nfs> ls . .. .trashcan fedora rhel-7 nfs> umountall Unmount `/repos' nfs> quit Close `storage.lan.nixpanic.net' You can also put the commands in a textfile and pipe that to nfsshell: $ cat /tmp/umountall-reproducer.nfsshell host storage.lan.example.net mount /repos ls umountall quit $ ./nfsshell < /tmp/umountall-reproducer.nfsshell nfs> host storage.lan.nixpanic.net Open storage.lan.example.net (172.31.96.2) TCP nfs> mount /repos Mount `/repos', TCP, transfer size 1048576 bytes. nfs> ls . .. .trashcan fedora rhel-7 nfs> umountall Unmount `/repos' nfs> quit Close `storage.lan.nixpanic.net' I'd appreciate it if someone can look into reproducing the problem and confirm the patch fixes it. Riyas, could you post the exact steps with nfsshell to reproduce the segfault? I'd like them to be recorded here as well. REVIEW: https://review.gluster.org/16625 (gNFS: Keep the mountdict as long as the service is active) posted (#1) for review on release-3.10 by Niels de Vos (ndevos) REVIEW: https://review.gluster.org/16626 (gNFS: Keep the mountdict as long as the service is active) posted (#1) for review on release-3.9 by Niels de Vos (ndevos) This bug is getting closed because a release has been made available that should address the reported issue. In case the problem is still not fixed with glusterfs-3.11.0, please open a new bug report. glusterfs-3.11.0 has been announced on the Gluster mailinglists [1], packages for several distributions should become available in the near future. Keep an eye on the Gluster Users mailinglist [2] and the update infrastructure for your distribution. [1] http://lists.gluster.org/pipermail/announce/2017-May/000073.html [2] https://www.gluster.org/pipermail/gluster-users/ |