Bug 1701077
| Summary: | NFS4 Server can crash whole server / cause data loss | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | Slawomir Pryczek <slawek1211> | ||||
| Component: | kernel | Assignee: | Kernel Maintainer List <kernel-maint> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Fedora Extras Quality Assurance <extras-qa> | ||||
| Severity: | urgent | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 29 | CC: | airlied, bskeggs, hdegoede, ichavero, itamar, jarodwilson, jeremy, jglisse, john.j5live, jonathan, josef, kernel-maint, linville, mchehab, mjg59, steved | ||||
| Target Milestone: | --- | ||||||
| Target Release: | --- | ||||||
| Hardware: | x86_64 | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | kernel-5.0.9-301.fc30 | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2019-04-25 19:33:39 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
|
Description
Slawomir Pryczek
2019-04-18 00:57:29 UTC
Some more info about the issue. Tested this with Debian 9 (Linux debian 4.9.0-8-amd64 #1 SMP), and it for sure seems kernel issue because there the bug manifests exactly as when we have used nfsv4 on Fedora 26. So instead of kernel errors / killing the machine - the nfs server is loosing ability to write anything to the share. After issuing "echo 123 > t.txt" the shell locks indefinitely, and some files could be read but not all (maybe because of some cache which is still working properly). This happened to us on production just couple of times...
The only error which is seen in dmesg is "nfs4_reclaim_open_state: Lock reclaim failed!". It'll kill nfs 4.2, 4.1. NFS 4.2 takes around 5-10s to be killed and 4.1 around a minute. After that the server needs restart /etc/init.d (stop/start/reload) seems to not have any effect.
Very strange thing is that if the client connects using nfs 4.0 then everything works fine ("mount -t nfs -o vers=4.0 127.0.0.1:/home /homenfs/") and there are no errors in dmesg. Maybe because it's slower because the server seems to be also harder to kill when enabling nfsd debugging...
Also this seems related: https://bugzilla.kernel.org/show_bug.cgi?id=115521
Another update on the issue. Tested this on Fedora 30 Server Edition (5.0.7-300.fc30.x86_64 #1 SMP) same thing happens. Posting test/exploit code and there are more logs on github (compilation instructions at the top) https://github.com/slawomir-pryczek/drbd_kill NFS 4.1 and 4.2 affected. NFS 4.0 and NFS 3 is working fine Reported that to kernel bugtracker... seems patch is ready https://bugzilla.kernel.org/show_bug.cgi?id=203363 kernel-tools-5.0.9-300.fc30 kernel-headers-5.0.9-300.fc30 kernel-5.0.9-300.fc30 has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-e84f6c34da kernel-tools-5.0.9-200.fc29 kernel-headers-5.0.9-200.fc29 kernel-5.0.9-200.fc29 has been submitted as an update to Fedora 29. https://bodhi.fedoraproject.org/updates/FEDORA-2019-1e8a4c6958 kernel-tools-5.0.9-100.fc28 kernel-headers-5.0.9-100.fc28 kernel-5.0.9-100.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2019-1b986880ea kernel-5.0.9-300.fc30, kernel-headers-5.0.9-300.fc30, kernel-tools-5.0.9-300.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-e84f6c34da kernel-5.0.9-100.fc28, kernel-headers-5.0.9-100.fc28, kernel-tools-5.0.9-100.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-1b986880ea kernel-5.0.9-200.fc29, kernel-headers-5.0.9-200.fc29, kernel-tools-5.0.9-200.fc29 has been pushed to the Fedora 29 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-1e8a4c6958 kernel-5.0.9-301.fc30 kernel-headers-5.0.9-300.fc30 kernel-tools-5.0.9-300.fc30 has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-e84f6c34da kernel-5.0.9-301.fc30, kernel-headers-5.0.9-300.fc30, kernel-tools-5.0.9-300.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-e84f6c34da kernel-5.0.9-200.fc29, kernel-headers-5.0.9-200.fc29, kernel-tools-5.0.9-200.fc29 has been pushed to the Fedora 29 stable repository. If problems still persist, please make note of it in this bug report. kernel-5.0.9-301.fc30, kernel-headers-5.0.9-300.fc30, kernel-tools-5.0.9-300.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report. kernel-5.0.9-100.fc28, kernel-headers-5.0.9-100.fc28, kernel-tools-5.0.9-100.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report. |