I'm able to reproduce the problem on your system with your binary but if I build QEMU from master on your system I'm not able to reproduce the problem. Could you provide your configure command line?
(In reply to Laurent Vivier from comment #1) > I'm able to reproduce the problem on your system with your binary but if I > build QEMU from master on your system I'm not able to reproduce the problem. > > Could you provide your configure command line? I'm not sure about Stefano, but I've just been using my distro qemu, specifically qemu-system-x86-6.2.0-14.fc36.x86_64. Maybe the bug has been fixed upstream already?
Sorry, I missed this comment: (In reply to Laurent Vivier from comment #1) > I'm able to reproduce the problem on your system with your binary but if I > build QEMU from master on your system I'm not able to reproduce the problem. > > Could you provide your configure command line? It's: ../configure --target-list=x86_64-softmmu based on 09ed077d7f plus the latest version of your AF_UNIX socket patchset. Feature summary (from meson.log): qemu 7.0.91 Directories Install prefix : /usr/local BIOS directory : share/qemu firmware path : share/qemu-firmware binary directory : /usr/local/bin library directory : /usr/local/lib/x86_64-linux-gnu module directory : lib/x86_64-linux-gnu/qemu libexec directory : /usr/local/libexec include directory : /usr/local/include config directory : /usr/local/etc local state directory : /var/local Manual directory : /usr/local/share/man Doc directory : /usr/local/share/doc Build directory : /home/sbrivio/qemu/build Source path : /home/sbrivio/qemu GIT submodules : ui/keycodemapdb tests/fp/berkeley-testfloat-3 tests/fp/berkeley-softfloat-3 dtc slirp Host binaries git : git make : make python : /usr/bin/python3 (version: 3.10) sphinx-build : NO gdb : /usr/bin/gdb iasl : NO genisoimage : smbd : /usr/sbin/smbd Configurable features Documentation : NO system-mode emulation : YES user-mode emulation : NO block layer : YES Install blobs : YES module support : NO fuzzing support : NO Audio drivers : oss Trace backends : log D-Bus display : NO QOM debugging : NO vhost-kernel support : YES vhost-net support : YES vhost-user support : YES vhost-user-crypto support : YES vhost-user-blk server support: YES vhost-vdpa support : YES build guest agent : YES Compilation host CPU : x86_64 host endianness : little C compiler : cc -m64 -mcx16 Host C compiler : cc -m64 -mcx16 C++ compiler : c++ -m64 -mcx16 CFLAGS : -O2 -g CXXFLAGS : -O2 -g QEMU_CFLAGS : -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -Wold-style-declaration -Wold-style-definition -Wtype-limits -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wempty-body -Wnested-externs -Wendif-labels -Wexpansion-to-defined -Wimplicit-fallthrough=2 -Wno-missing-include-dirs -Wno-shift-negative-value -Wno-psabi -fstack-protector-strong QEMU_CXXFLAGS : -D__STDC_LIMIT_MACROS -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wundef -Wwrite-strings -fno-strict-aliasing -fno-common -fwrapv -Wtype-limits -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wempty-body -Wendif-labels -Wexpansion-to-defined -Wimplicit-fallthrough=2 -Wno-missing-include-dirs -Wno-shift-negative-value -Wno-psabi -fstack-protector-strong QEMU_OBJCFLAGS : -Wold-style-definition -Wtype-limits -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wempty-body -Wnested-externs -Wendif-labels -Wexpansion-to-defined -Wno-initializer-overrides -Wno-missing-include-dirs -Wno-shift-negative-value -Wno-string-plus-int -Wno-typedef-redefinition -Wno-tautological-type-limit-compare -Wno-psabi QEMU_LDFLAGS : -Wl,--warn-common -Wl,-z,relro -Wl,-z,now -fstack-protector-strong profiler : NO link-time optimization (LTO) : NO PIE : YES static build : NO malloc trim support : YES membarrier : NO debug stack usage : NO mutex debugging : NO memory allocator : system avx2 optimization : YES avx512f optimization : NO gprof enabled : NO gcov : NO thread sanitizer : NO CFI support : NO strip binaries : NO sparse : NO mingw32 support : NO Cross compilers x86_64 : cc Targets and accelerators KVM support : YES HAX support : NO HVF support : NO WHPX support : NO NVMM support : NO Xen support : NO TCG support : YES TCG backend : native (x86_64) TCG plugins : YES TCG debug enabled : NO target list : x86_64-softmmu default devices : YES out of process emulation : YES vfio-user server : NO Block layer support coroutine backend : ucontext coroutine pool : YES Block whitelist (rw) : Block whitelist (ro) : Use block whitelist in tools : NO VirtFS support : YES build virtiofs daemon : YES Live block migration : YES replication support : YES bochs support : YES cloop support : YES dmg support : YES qcow v1 support : YES vdi support : YES vvfat support : YES qed support : YES parallels support : YES FUSE exports : NO VDUSE block exports : YES Crypto TLS priority : NORMAL GNUTLS support : NO libgcrypt : NO nettle : NO AF_ALG support : NO rng-none : NO Linux keyring : YES Dependencies SDL support : NO SDL image support : NO GTK support : NO pixman : YES 0.40.0 VTE support : NO slirp support : YES 4.7.0 libtasn1 : NO PAM : NO iconv support : YES curses support : YES virgl support : NO curl support : NO Multipath support : NO PNG support : YES 1.6.37 VNC support : YES VNC SASL support : NO VNC JPEG support : YES 2.1.2 OSS support : YES ALSA support : YES 1.2.6.1 PulseAudio support : NO JACK support : NO brlapi support : NO vde support : NO netmap support : NO l2tpv3 support : YES Linux AIO support : NO Linux io_uring support : NO ATTR/XATTR support : YES RDMA support : NO PVRDMA support : NO fdt support : internal libcap-ng support : YES bpf support : NO spice protocol support : NO rbd support : NO smartcard support : NO U2F support : NO libusb : NO usb net redir : NO OpenGL support (epoxy) : NO GBM : NO libiscsi support : NO libnfs support : NO seccomp support : YES 2.5.4 GlusterFS support : NO TPM support : YES libssh support : NO lzo support : NO snappy support : NO bzip2 support : NO lzfse support : NO zstd support : NO NUMA host support : YES capstone : NO libpmem support : NO libdaxctl support : NO libudev : NO FUSE lseek : NO selinux : YES 3.3 Subprojects libvduse : YES libvhost-user : YES User defined options Native files : /home/sbrivio/qemu/build/config-meson.cross backend : ninja prefix : /usr/local werror : true Note that there are two binaries installed on my system: one if from distribution package (based on 7.0.0) in /usr/bin, the other one I described above is in /usr/local/bin (taking precedence). Would it make sense that I try and rebase the version under /usr/local/bin/ to latest upstream?
From a previous analysis (Stefano?): > virtio_net_tx_bh() calls virtio_net_flush_tx() > > - if qemu_sendv_packet_async() returns 0 (where returning 0 > is a summary for a number of conditions...), which for some reason > happens less frequently with a higher tx_burst, virtio_net_flush_tx() > returns -EBUSY > > - in that case, virtio_net_tx_bh() doesn't reschedule. Sure, as the > comment says, notifications are re-enabled by the callback for > qemu_sendv_packet_async(), i.e. virtio_net_tx_complete(), which > however doesn't take care of rescheduling > > - ...not even if we were in the middle of a "burst", > virtio_net_tx_complete() does its job and calls virtio_net_flush_tx() > one last time, and that's it. Perhaps it's fine in most cases, I'm > not sure why I don't hit the stall reliably on -EBUSY from > virtio_net_flush_tx(). In the case of net/socket.c (net/stream.c with my patch series) the only reason qemu_sendv_packet_async() sends back 0 is when net_socket_receive() (net_stream_receive()) cannot send the full buffer. A quick hack (with stream.c) that retries to send the buffer rather than to return 0 seems to fix the problem: diff --git a/net/stream.c b/net/stream.c index a84295f209..3ab2966bb9 100644 --- a/net/stream.c +++ b/net/stream.c @@ -85,7 +85,7 @@ static ssize_t net_stream_receive(NetClientState *nc, const uint8_t *buf, size_t remaining; ssize_t ret; - +again: remaining = iov_size(iov, 2) - s->send_index; nlocal_iov = iov_copy(local_iov, 2, iov, 2, s->send_index, remaining); ret = qio_channel_writev(s->ioc, local_iov, nlocal_iov, NULL); @@ -98,6 +98,7 @@ static ssize_t net_stream_receive(NetClientState *nc, const uint8_t *buf, } if (ret < (ssize_t)remaining) { s->send_index += ret; + goto again; s->ioc_write_tag = qio_channel_add_watch(s->ioc, G_IO_OUT, net_stream_writable, s, NULL); return 0; I'm investigating why net_socket_writable() (net_stream_writable()) doesn't send the remaining data of the buffer.
Setting vectors parameter of virtio-net device to 1 seems to be another workaround: ... -device virtio-net-pci,vectors=1,...
(In reply to Laurent Vivier from comment #5) > Setting vectors parameter of virtio-net device to 1 seems to be another > workaround: > > ... -device virtio-net-pci,vectors=1,... I enthusiastically tried that to replace the current workaround in passt's testsuite, but I just hit the issue (just once) even with that. It doesn't happen as frequently, though, it might be a different issue, and I didn't have means to collect more information about it on the spot.
Wenli Are you able to reproduce in QE's lab?
(In reply to Chao Yang from comment #7) > Wenli > > Are you able to reproduce in QE's lab? I tried 10 times with 9.2 downstream version qemu-kvm-7.1.0-1.el9.x86_64/5.14.0-70.13.1.el9_0.x86_64(host), can not reproduce issue.
This is now fixed in upstream qemu by: commit df8d07081718c29d04d106583d9c300128686cda Author: Laurent Vivier <lvivier> Date: Thu Oct 20 11:58:45 2022 +0200 virtio-net: fix bottom-half packet TX on asynchronous completion and it will make its way into RHEL 9.2 together with the planned rebase to qemu 7.2: https://bugzilla.redhat.com/show_bug.cgi?id=2135806. Setting this ticket as blocked by that one.
Updating the rest of the fields necessary for inclusion by rebase
QE bot(pre verify): Set 'Verified:Tested,SanityOnly' as gating/tier1 test pass.
Test with qemu-kvm-7.2.0-1.el9 10 times with steps from comment #0, the iperf3 works well and perforamnce looks good. so set it to VERIFIED. [root@dell-per440-18 ~]# iperf3 -c ${gw} -Z -P 5 -l 1M -i1 -t30 -O5 --pacing-timer 100000 - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-30.00 sec 13.8 GBytes 3.94 Gbits/sec 0 sender [ 5] 0.00-30.01 sec 13.8 GBytes 3.94 Gbits/sec receiver [ 7] 0.00-30.00 sec 13.7 GBytes 3.93 Gbits/sec 0 sender [ 7] 0.00-30.01 sec 13.7 GBytes 3.92 Gbits/sec receiver [ 9] 0.00-30.00 sec 13.7 GBytes 3.91 Gbits/sec 0 sender [ 9] 0.00-30.01 sec 13.7 GBytes 3.91 Gbits/sec receiver [ 11] 0.00-30.00 sec 13.6 GBytes 3.90 Gbits/sec 0 sender [ 11] 0.00-30.01 sec 13.6 GBytes 3.90 Gbits/sec receiver [ 13] 0.00-30.00 sec 13.3 GBytes 3.81 Gbits/sec 0 sender [ 13] 0.00-30.01 sec 13.3 GBytes 3.81 Gbits/sec receiver [SUM] 0.00-30.00 sec 68.0 GBytes 19.5 Gbits/sec 0 sender [SUM] 0.00-30.01 sec 68.1 GBytes 19.5 Gbits/sec receiver
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: qemu-kvm security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2023:2162