Thanks for filing it; yes we should look at this sometime, while being slower than a block device doesn't surprise me, we should
be able to find a way to match NFS; conceptually they're similar (stuff file operations into a protocol, shuffle them
over a channel and handle them in a host daemon).

Running slower than nfs is definitely surprising. I have yet to setup, NFS for my VM. I am experimenting with different configurations of virtio-fs and trying to figure out which runs fastest for this workload. With cache=auto, I am seeing a runtime of around 4 and with DAX enabled it came close to 3.5 minutes. So some improvement. Note, DAX support is in upstream kernel but not in qemu yet. It is still being worked on.

cache=auto, xattr=off,thread-pool-size=0
----------------------------------------

real    4m4.506s
user    19m0.169s
sys     26m21.216s

cache=auto, dax=on, xattr=off, thrad-pool-size=0
------------------------------------------------
real    3m36.421s
user    18m55.979s
sys     24m2.325s


Maybe NFS is cutting down on number of round trips (and not per round trip time) and probably that helps with speed. This will require more investigation.

And tried with nfs now. It is faster. Results vary significantly from run to run. Here are results of 3 runs.

real    3m7.838s
user    18m51.170s
sys     15m51.583s

real	3m3.128s
user	18m51.658s
sys	15m51.310s

real	2m46.605s
user	18m38.966s
sys	15m27.063s

So yes, nfs seems faster... Using DAX with virtiofs reduces the gap, but still nfs is faster.

Looks like, virtiofs is not too bad if build output is going back over filesystem as well. IOW, so far we are testing with O=/tmp/build. Alex, try sending writes back over filesystem. O=/mnt/nfs/linux/build/. I notice that difference is not as much any more. I suspect we are not as efficient on read path as we could have been.

In fact if I test this with DAX enabled, I see numbers better than nfs. So here are the numbers with build output going back to host.

nfs
===
real    4m6.918s
user    19m24.189s
sys     17m12.816s

real    4m3.101s
user    19m17.690s
sys     16m52.923s

real    4m10.282s
user    19m22.076s
sys     16m56.548s

virtiofs (cache=auto, -o xattr, --thread-pool-size=0)
=====================================================

real    4m34.014s
user    18m18.311s
sys     19m12.187s

real    4m25.076s
user    18m24.972s
sys     19m14.242s

real    4m16.685s
user    18m27.491s
sys     19m8.420s

virtiofs (cache=auto, -o xattr, --thread-pool-size=0, dax enabled)
==================================================================
real    3m26.864s
user    18m29.511s
sys     16m42.756s

real    3m31.442s
user    18m25.636s
sys     16m44.484s

real    3m26.130s
user    18m25.922s
sys     16m40.312s


So without DAX, difference between nfs and virtiofs is not as much. And with DAX, virtiofs is performing better for this particular workload/setting.

TBH, this looks like you're slowing down nfs to show that virtiofs can be proportionally less worse.

(In reply to Alex Williamson from comment #5)
> TBH, this looks like you're slowing down nfs to show that virtiofs can be
> proportionally less worse.

I think a more fair characterization will be that with a slightly different workload, virtiofs is faster. So our task at hand will be to see if we can improve performance of virtiofs for the workload you threw at it.

One thing I remembered is that while virtiofsd is a user space process, NFS is done in the kernel - I don't know what % of the benefit comes from that, but it's
something to keep in mind.

One thing that NFS server allows is delegations: it's a kind lease granted to the client.  If a client receives a read/(write) delegations it can open/read/stat/(write/chmod/...) the file without going out to the network.

The downside for virtualized clients is that a broken client can DoS the server, though I guess there are timeouts involved, as with other NFS operations.

It would be interesting to compare the number of roundtrips made by NFS and virtiofs for a certain workload.  RTT statistics would also be interesting.

@Yanhui,hi
I see this bug is about performance and we don't cover it in our test, can you take it?

I'm going to assign this to Vivek and raise the severity and priority, given that I think this needs to be at least well understood before we can generally support virtiofs...

(In reply to Klaus Heinrich Kiwi from comment #11)
> I'm going to assign this to Vivek and raise the severity and priority, given
> that I think this needs to be at least well understood before we can
> generally support virtiofs...

Vivek,

 have you had a chance to check if we're any better with the latest iterations? I've tagged this with RFE as I don't think it's a blocking bug for RHEL 9 GA, but let me know if we need to discuss.

(In reply to Klaus Heinrich Kiwi from comment #12)
> (In reply to Klaus Heinrich Kiwi from comment #11)
> > I'm going to assign this to Vivek and raise the severity and priority, given
> > that I think this needs to be at least well understood before we can
> > generally support virtiofs...
> 
> Vivek,
> 
>  have you had a chance to check if we're any better with the latest
> iterations? I've tagged this with RFE as I don't think it's a blocking bug
> for RHEL 9 GA, but let me know if we need to discuss.

Hi Klaus,

I don't think this is a blocking bug. There is no hard requirement that virtiofs has to be faster than NFS in all cases. It will be nice though.

Have not got a chance to test it on latest configurations. In general, I have an internal TODO item to do some investigation and see how virtiofs performance can be improved.

One of the things I was thinking what if we run virtiofs as part of qemu (And not as a separate vhost-user device). Can that give us some performance boost.

Another thing we are relying on is DAX support in qemu. Which can give us performance boost.

Vivek

Hello Vivek,
This bug will be auto-close,if you want to keep it open could you pls extend the 'stale date'?

Regards,
Xiaoling

Hi Xiaoling,

I think we can let this bug auto close. Improving performance of virtiofs is an ongoing item. DAX support (qemu side) is in progress as well. So its not one thing which needs to be done. Over a period of time our target is to keep on improving performance of virtiofs when we identify a bottleneck.

I consider this to be more of a desirable property of virtiofs (and any software component in general) and not necessarily a bug. 

So I think its better that this bug is closed.

Vivek

After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.