Bug 2127063
| Summary: | [RHEL8.7] virtio-nfs generic/694 failure: rm: cannot remove '/mnt/test/694': Directory not empty | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | Hu Shuai (Fujitsu) <hshuai> |
| Component: | qemu-kvm | Assignee: | German Maglione <gmaglione> |
| qemu-kvm sub component: | virtio-fs | QA Contact: | Hu Shuai (Fujitsu) <hshuai> |
| Status: | CLOSED CANTFIX | Docs Contact: | |
| Severity: | unspecified | ||
| Priority: | unspecified | CC: | coli, dgilbert, vgoyal, virt-maint, xiagao, yidliu |
| Version: | 8.7 | Keywords: | Triaged |
| Target Milestone: | rc | Flags: | pm-rhel:
mirror+
|
| Target Release: | --- | ||
| Hardware: | All | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-10-11 09:31:27 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 2089955 | ||
This can also be reproduced on RHEL8.6
Test Env:
Host kernel: kernel-4.18.0-372.9.1.el8.aarch64
libvirt-8.0.0-5.module+el8.6.0+14480+c0a3aa0f.aarch64
qemu-kvm-6.2.0-11.module+el8.6.0+14707+5aa4b42d.aarch64
Test Result:
```
# ./check -virtiofs generic/694
FSTYP -- virtiofs
PLATFORM -- Linux/aarch64 localhost 4.18.0-372.9.1.el8.aarch64 #1 SMP Fri Apr 15 22:01:11 EDT 2022
MKFS_OPTIONS -- nfsmyfs1
MOUNT_OPTIONS -- -o context=system_u:object_r:root_t:s0 nfsmyfs1 /mnt/scratch
generic/694 118s ... - output mismatch (see /root/xfstests/results//generic/694.out.bad)
--- tests/generic/694.out 2022-09-16 12:48:22.930370350 +0800
+++ /root/xfstests/results//generic/694.out.bad 2022-09-16 13:46:46.353646740 +0800
@@ -1,2 +1,3 @@
QA output created by 694
Silence is golden
+rm: cannot remove '/mnt/test/694': Directory not empty
...
(Run 'diff -u /root/xfstests/tests/generic/694.out /root/xfstests/results//generic/694.out.bad' to see the entire diff)
Ran: generic/694
Failures: generic/694
Failed 1 of 1 tests
```
x86 also has this issue on RHEL870, and passed on RHEL910. (In reply to xiagao from comment #3) > x86 also has this issue on RHEL870, and passed on RHEL910. In RHEL9.1 are you using the rust version? With or without --inode-fiel-handle=prefer|mandatory? (In reply to German Maglione from comment #4) > (In reply to xiagao from comment #3) > > x86 also has this issue on RHEL870, and passed on RHEL910. > > In RHEL9.1 are you using the rust version? With or without > --inode-fiel-handle=prefer|mandatory? Yes, I use rust version in RHEL9.1 and didn't use --inode-fiel-handle=prefer|mandatory . virtiofsd version: virtiofsd-1.4.0-1.el9.x86_64 virtiofsd cmdline: /usr/libexec/virtiofsd --socket-path=/var/tmp/avocado_gecuen7e/avocado-vt-vm1-fs1-virtiofsd.sock -o source=/tmp/virtio_fs1_test --no-killpriv-v2 -o cache=auto. This cannot be solved in virtiofsd, the solution is to: - that the NFS server implements OPEN4_RESULT_PRESERVE_UNLINKED, or server-side silly rename. - or add fuse support for synchronous forgets The current workaround is to run virtiofsd with `--inode-file-handles=mandatory`, but this has its own limitations, for instance, requiring `CAP_DAC_READ_SEARCH`. I have opened an upstream issue to keep track of this bug. https://gitlab.com/virtio-fs/virtiofsd/-/issues/61 More info about silly rename: - [Server-side silly rename](https://linux-nfs.org/wiki/index.php/Server-side_silly_rename) - [Linux NFS FAQ](https://nfs.sourceforge.net/) D2. What is a "silly rename"? Why do these .nfsXXXXX files keep showing up? Hello German
> The current workaround is to run virtiofsd with `--inode-file-handles=mandatory`,
> but this has its own limitations, for instance, requiring `CAP_DAC_READ_SEARCH`.
There is no `--inode-file-handles` parameter for virtiofsd for RHEL8.7/RHEL8.8 aarch64,
so I can not verify this workaround.
(In reply to Hu Shuai (Fujitsu) from comment #7) > Hello German > > > The current workaround is to run virtiofsd with `--inode-file-handles=mandatory`, > > but this has its own limitations, for instance, requiring `CAP_DAC_READ_SEARCH`. > > There is no `--inode-file-handles` parameter for virtiofsd for > RHEL8.7/RHEL8.8 aarch64, > so I can not verify this workaround. Sorry, I forgot to mention that `--inode-file-handles=mandatory` is only available in the Rust version of virtiofsd. We are looking at how to solve it from fuse. |
Description of problem: xfstests generic/694 fails with virtiofs + nfs on RHEL8.7 Nightly build. Strangely, it happens almost only the first time the generic/694 test is executed on the guest, especially when the value of the virtiofsd parameter --thread-pool-size is set to a value greater than 1 Version-Release number of selected component (if applicable): Red Hat Enterprise Linux Version Number: RHEL8 Release Number: RHEL-8.7.0-20220912.0 Architecture: aarch64 Kernel Version: kernel-4.18.0-425.el8.aarch64 Related Package Version: libvirt-8.0.0-10.module+el8.7.0+16047+746a126c.aarch64 qemu-kvm-6.2.0-20.module+el8.7.0+16496+35f7e655.aarch64 How reproducible: almost 100% at the first execution on the guest with `--thread-pool-size=3` for virtiofsd Steps to Reproduce: 1. Prepare guest with `--thread-pool-size=3` for virtiofsd 2. Run generic/694 of xfstests with using virtiofs + nfs filesystem for the test directory. # ./check -virtiofs generic/694 Actual results: ``` # ./check -virtiofs generic/694 FSTYP -- virtiofs PLATFORM -- Linux/aarch64 localhost 4.18.0-425.el8.aarch64 #1 SMP Fri Sep 9 12:37:07 EDT 2022 MKFS_OPTIONS -- nfsmyfs1 MOUNT_OPTIONS -- -o context=system_u:object_r:root_t:s0 nfsmyfs1 /mnt/scratch generic/694 107s ... - output mismatch (see /root/xfstests/results//generic/694.out.bad) --- tests/generic/694.out 2022-09-15 13:27:01.336767060 +0800 +++ /root/xfstests/results//generic/694.out.bad 2022-09-15 16:22:25.268529210 +0800 @@ -1,2 +1,3 @@ QA output created by 694 Silence is golden +rm: cannot remove '/mnt/test/694': Directory not empty ... (Run 'diff -u /root/xfstests/tests/generic/694.out /root/xfstests/results//generic/694.out.bad' to see the entire diff) Ran: generic/694 Failures: generic/694 Failed 1 of 1 tests ``` Expected results: PASS Additional info: