German,

 will 'soft-assign' this to you, but it's medium priority - so that you can take your time exploring and learning.

Hanna cc'ed fyi only

Also hit this issue on RHEL8.6.0(host + guest)
pkg:
qemu-kvm-6.1.0-4.module+el8.6.0+13039+4b81a1dc.x86_64
kernel-4.18.0-348.4.el8.x86_64(host and guest)

I was not able to reproduce the error:

host: Fedora 34
    Linux fedora 5.14.13-200.fc34.x86_64 #1 SMP Mon Oct 18 12:39:31 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

guest: Fedora 35 Server
    Linux ibm-p8-kvm-03-guest-02.virt.pnr.lab.eng.rdu2.redhat.com 5.14.10-300.fc35.x86_64 #1 SMP Thu Oct 7 20:48:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

(I had to increase the limit of open files in both the guest and the host to avoid "Too many open files" error)

with both:
    QEMU emulator version 6.1.50 (v6.1.0-1605-gafc9fcde55)
    QEMU emulator version 5.2.0 (qemu-5.2.0-8.fc34)

qemu options:    
    -blockdev file,node-name=hdd,filename=fedora-srv.raw \
    -device virtio-blk,drive=hdd \
    -accel kvm -m 4G \
    -chardev socket,id=char_virtiofs_fs1,path=socket_path1 \
    -device vhost-user-fs-pci,id=vufs_virtiofs_fs1,chardev=char_virtiofs_fs1,tag=myfs1,queue-size=1024 \
    -chardev socket,id=char_virtiofs_fs2,path=socket_path2 \
    -device vhost-user-fs-pci,id=vufs_virtiofs_fs2,chardev=char_virtiofs_fs2,tag=myfs2,queue-size=1024 \
    -object memory-backend-file,id=mem,size=4G,mem-path=/dev/shm,share=on \
    -numa node,memdev=mem
    

using the virtiofsd-rs: https://gitlab.com/virtio-fs/virtiofsd-rs
with a combination of the following parameters (with and without):
    --cache always --xattr --announce-submounts --inode-file-handles 
    and both --sandbox namespace  and --sandbox none    

Results:
FSTYP         -- virtiofs
PLATFORM      -- Linux/x86_64 ibm-p8-kvm-03-guest-02 5.14.10-300.fc35.x86_64 #1 SMP Thu Oct 7 20:48:44 UTC 2021
MKFS_OPTIONS  -- myfs1
MOUNT_OPTIONS -- myfs1 /mnt/myfs1

generic/006 6s ...  6s
Ran: generic/006
Passed all 1 tests

(the times varies between 6 and 7s)

Xiagao, could you re-run your tests using virtiofsd-rs? https://gitlab.com/virtio-fs/virtiofsd-rs

Hi,

I took a look and here’s what I found:

First, as shown by the `date` invocations, the reproducer takes very long.  In comment 6, it takes over one hour, in comment 0 it takes 40 minutes.  I find it’s reproducible much more quickly by drastically reducing the number of virtual cores from 24 to 2, by using `-smp 2,maxcpus=2,cores=1,threads=1,dies=1,sockets=2`.

Second, the reproducing steps in comment 0 say to run `./check -virtiofs generic/006` (step 4), but this BZ is actually about generic/650, as shown under “Actual results”.  Consequentially, German tested generic/006 in comment 3, but it should be generic/650.

Third, I can reproduce the problem (as stated above already with two CPUs on two sockets with one core each), and I believe it’s NFS-related.  I remember that NFS creates temporary lock files for each open FD, and these cannot be deleted, hence the failure output.  I’d have to dig what our approach was to this, but AFAIR we treated it mostly as an unfortunate but not directly fixable problem.

However, this problem can be circumvented by not opening FDs for lookups, and that’s precisely what the `--inode-file-handles` option of virtiofsd-rs is for.  With this option, the test does indeed pass for me.  It fails without this option.  I don’t know why the problem persists in comment 6 despite passing the option, but it’s notable that it’s a best-effort option: If file handles don’t work for a filesystem, then we fall back to using FDs for lookups.  I don’t know why file handles wouldn’t work here, though.  Perhaps it’s the underlying filesystem on the NFS host that doesn’t support file handles, but I was under the impression that NFS requires file handles – I may well be wrong, though.

Vivek once suggested that with `--inode-file-handles` we should verify whether the shared directory’s root filesystem supports file handles and return an error if it doesn’t.  I’m not sure I agree because it still wouldn’t fix the problem; virtiofsd-rs would just refuse to start up, and then you’d drop the option, and then the problem of these NFS hidden files would persist.

Hanna

(In reply to Hanna Reitz from comment #7)
> Hi,
> 
> I took a look and here’s what I found:
> 
> First, as shown by the `date` invocations, the reproducer takes very long. 
> In comment 6, it takes over one hour, in comment 0 it takes 40 minutes.  I
> find it’s reproducible much more quickly by drastically reducing the number
> of virtual cores from 24 to 2, by using `-smp
> 2,maxcpus=2,cores=1,threads=1,dies=1,sockets=2`.

Yes, generic/650 is testing vcpu hotplugging, so the running time depends on
the vcpu numbers.
This is the case description.
# Run an all-writes fsstress run with multiple threads while exercising CPU
# hotplugging to shake out bugs in the write path.


> 
> Second, the reproducing steps in comment 0 say to run `./check -virtiofs
> generic/006` (step 4), but this BZ is actually about generic/650, as shown
> under “Actual results”.  Consequentially, German tested generic/006 in
> comment 3, but it should be generic/650.
It's my fault, I'm very sorry to paste wrong test case id.It should be generic/650.

> 
> Third, I can reproduce the problem (as stated above already with two CPUs on
> two sockets with one core each), and I believe it’s NFS-related.  I remember
> that NFS creates temporary lock files for each open FD, and these cannot be
> deleted, hence the failure output.  I’d have to dig what our approach was to
> this, but AFAIR we treated it mostly as an unfortunate but not directly
> fixable problem.
> 
> However, this problem can be circumvented by not opening FDs for lookups,
> and that’s precisely what the `--inode-file-handles` option of virtiofsd-rs
> is for.  With this option, the test does indeed pass for me.  It fails
> without this option.  I don’t know why the problem persists in comment 6
> despite passing the option, but it’s notable that it’s a best-effort option:
> If file handles don’t work for a filesystem, then we fall back to using FDs
> for lookups.  I don’t know why file handles wouldn’t work here, though. 
> Perhaps it’s the underlying filesystem on the NFS host that doesn’t support
> file handles, but I was under the impression that NFS requires file handles
> – I may well be wrong, though.

I tested it again according to comment6,it passed.
[root@bootp-73-75-204 xfstests-dev]# date; ./check -virtiofs generic/650; date
Wed Nov 17 10:37:31 AM CST 2021
FSTYP         -- virtiofs
PLATFORM      -- Linux/x86_64 bootp-73-75-204 5.14.0-13.el9.x86_64 #1 SMP Mon Nov 8 20:07:06 EST 2021
MKFS_OPTIONS  -- myfs2
MOUNT_OPTIONS -- -o context=system_u:object_r:root_t:s0 myfs2 /mnt/myfs2

generic/650 192s ...  190s
Ran: generic/650
Passed all 1 tests

Wed Nov 17 10:40:41 AM CST 2021


Thanks.

> 
> Vivek once suggested that with `--inode-file-handles` we should verify
> whether the shared directory’s root filesystem supports file handles and
> return an error if it doesn’t.  I’m not sure I agree because it still
> wouldn’t fix the problem; virtiofsd-rs would just refuse to start up, and
> then you’d drop the option, and then the problem of these NFS hidden files
> would persist.
> 
> Hanna

(In reply to Hanna Reitz from comment #7)
> Third, I can reproduce the problem (as stated above already with two CPUs on
> two sockets with one core each), and I believe it’s NFS-related.  I remember
> that NFS creates temporary lock files for each open FD, and these cannot be
> deleted, hence the failure output.  I’d have to dig what our approach was to
> this, but AFAIR we treated it mostly as an unfortunate but not directly
> fixable problem.

Yes, you are right (idk how I missed those .nfsxxxx files) is an issue with the NFS client.
This happens when an open file is unlinked, it is called "silly rename"
(I thought it was resolved in NFSv4).

Following is another open bug and I think it is similar. They all probably are related issues w.r.t NFS creating temporary .nfsxxxx files and unlink related races related to that.

https://bugzilla.redhat.com/show_bug.cgi?id=1908490

For the time being I think issue could be mitigated by using file handles (--inode-file-handles) in rust version of virtiofsd.

Longer term, we need to look into the idea of synchronous forget (I think david gilbert suggested that). Somebody needs to dive deeper and see if that is feasible or not.

Ideally, it would be nice if NFS can fix it and move away from these ".nfsxxxx" files. That will fix the issue.

Hi Vivek, do we have any plan for this bug?

Also I just test it with --inode-file-handles, the result is pass.

2022-09-08 03:53:38: generic/650
2022-09-08 04:36:30:  2572s

virtiofsd cmd:
/usr/libexec/virtiofsd --socket-path=/var/tmp/avocado_4psduw_6/avocado-vt-vm1-fs2-virtiofsd.sock -o source=/tmp/virtio_fs2_test --no-killpriv-v2 -o cache=auto --inode-file-handles=prefer.


virtiofsd version:
virtiofsd-1.4.0-1.el9.x86_64

(In reply to xiagao from comment #12)
> Hi Vivek, do we have any plan for this bug?

No updates on this. I don't think NFS has fixed this issue. So what we have currently is just the workaround of using --inode-file-handles.

BTW, I noticed that in another bug, it was claimed that tests passed on nfsv4 (but failed on nfsv3). Does that mean issue does not happen on nfsv4 and is limited to nfsv3 only? That kind of sounds strange.

(In reply to Vivek Goyal from comment #14)
> (In reply to xiagao from comment #12)
> > Hi Vivek, do we have any plan for this bug?
> 
> No updates on this. I don't think NFS has fixed this issue. So what we have
> currently is just the workaround of using --inode-file-handles.
Yes, I just have a test with and without "--inode-file-handles", the results show that it can pass with "--inode-file-handles" and will fail without this option.

> 
> BTW, I noticed that in another bug, it was claimed that tests passed on
> nfsv4 (but failed on nfsv3). Does that mean issue does not happen on nfsv4
> and is limited to nfsv3 only? That kind of sounds strange.
I thought you mean bz1908490.
From my test results recently, generic/245 always passed on RHEL9 on top of nfs version4.2,and I just try it on top of nfs version3, it also passed!

my nfs mount info:
127.0.0.1:/mnt/virtio_fs1_test_nfs on /tmp/virtio_fs1_test type nfs (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=127.0.0.1,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=127.0.0.1)
127.0.0.1:/mnt/virtio_fs2_test_nfs on /tmp/virtio_fs2_test type nfs (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=127.0.0.1,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=127.0.0.1)

xfstest generic/245 results: Pass

pkg:
KVM version: qemu-kvm-7.1.0-1.el9.x86_64
kernel version: 5.14.0-163.el9.x86_64(host && guest)

(In reply to xiagao from comment #15)
> (In reply to Vivek Goyal from comment #14)
> > (In reply to xiagao from comment #12)
> > > Hi Vivek, do we have any plan for this bug?
> > 
> > No updates on this. I don't think NFS has fixed this issue. So what we have
> > currently is just the workaround of using --inode-file-handles.
> Yes, I just have a test with and without "--inode-file-handles", the results
> show that it can pass with "--inode-file-handles" and will fail without this
> option.
> 
> > 
> > BTW, I noticed that in another bug, it was claimed that tests passed on
> > nfsv4 (but failed on nfsv3). Does that mean issue does not happen on nfsv4
> > and is limited to nfsv3 only? That kind of sounds strange.

Indeed, since the silly rename is still present in the latest nfs.
The required option (OPEN4_RESULT_PRESERVE_UNLINKED) was added to the 4.1 nfs spec,
but currently it's not implemented. I check the kernel source code (and test just in case)


> I thought you mean bz1908490.
> From my test results recently, generic/245 always passed on RHEL9 on top of
> nfs version4.2,and I just try it on top of nfs version3, it also passed!
> 
> my nfs mount info:
> 127.0.0.1:/mnt/virtio_fs1_test_nfs on /tmp/virtio_fs1_test type nfs
> (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,
> timeo=600,retrans=2,sec=sys,mountaddr=127.0.0.1,mountvers=3,mountport=20048,
> mountproto=udp,local_lock=none,addr=127.0.0.1)
> 127.0.0.1:/mnt/virtio_fs2_test_nfs on /tmp/virtio_fs2_test type nfs
> (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,
> timeo=600,retrans=2,sec=sys,mountaddr=127.0.0.1,mountvers=3,mountport=20048,
> mountproto=udp,local_lock=none,addr=127.0.0.1)
> 
> xfstest generic/245 results: Pass
> 
> pkg:
> KVM version: qemu-kvm-7.1.0-1.el9.x86_64
> kernel version: 5.14.0-163.el9.x86_64(host && guest)

This cannot be solved in virtiofsd, the solution is to:

- that the NFS server implements OPEN4_RESULT_PRESERVE_UNLINKED, or
  server-side silly rename.
  
- or add fuse support for synchronous forgets

 
The current workaround is to run virtiofsd with `--inode-file-handles=mandatory`,
but this has its own limitations, for instance, requiring `CAP_DAC_READ_SEARCH`.

I have opened an upstream issue to keep track of this bug.  
https://gitlab.com/virtio-fs/virtiofsd/-/issues/61  

More info about silly rename:
- [Server-side silly rename](https://linux-nfs.org/wiki/index.php/Server-side_silly_rename)
- [Linux NFS FAQ](https://nfs.sourceforge.net/)
  D2. What is a "silly rename"? Why do these .nfsXXXXX files keep showing up?