Description of problem: Boot dpdk's testpmd with vhost-user client, then start VM with vhost-user server fails with error: # virsh start rhel8.4 error: Failed to start domain rhel8.4 error: Cannot find 'ovs-vsctl' in path: No such file or directory Version-Release number of selected component (if applicable): 4.18.0-269.el8.x86_64 qemu-kvm-5.2.0-2.scrmod+el8.4.0+9260+5df81558.wrb201230.x86_64 libvirt-6.10.0-1.module+el8.4.0+8898+a84e86e1.x86_64 How reproducible: 100% Steps to Reproduce: 1. In host, boot dpdk's testpmd with vhost-user client sockets dpdk-testpmd \ -l 2,4,6,8,10,12,14,16,18 \ --socket-mem 1024,1024 \ -n 4 \ --vdev 'net_vhost0,iface=/tmp/vhost-user1,queues=2,client=1,iommu-support=1' \ --vdev 'net_vhost1,iface=/tmp/vhost-user2,queues=2,client=1,iommu-support=1' \ -d /usr/lib64/librte_net_vhost.so \ -- \ --portmask=f \ -i \ --rxd=512 --txd=512 \ --rxq=2 --txq=2 \ --nb-cores=8 \ --forward-mode=io 2. Boot VM with vhost-user server, fails <interface type='vhostuser'> <mac address='88:66:da:5f:dd:12'/> <source type='unix' path='/tmp/vhost-user1' mode='server'/> <model type='virtio'/> <driver name='vhost' queues='2' rx_queue_size='1024' iommu='on' ats='on'/> <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </interface> <interface type='vhostuser'> <mac address='88:66:da:5f:dd:13'/> <source type='unix' path='/tmp/vhost-user2' mode='server'/> <model type='virtio'/> <driver name='vhost' queues='2' rx_queue_size='1024' iommu='on' ats='on'/> <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/> </interface> # virsh start rhel8.4 error: Failed to start domain rhel8.4 error: Cannot find 'ovs-vsctl' in path: No such file or directory Actual results: Boot vhost-user ports fails. Expected results: Boot vhost-user ports should work well. Additional info: 1. This is a libvirt regression bug. libvirt-6.0.0-30.module+el8.4.0+8705+34397d87.x86_64 works well libvirt-6.6.0-8.module+el8.4.0+8855+a9e237a9.x86_64 works well libvirt-6.10.0-1.module+el8.4.0+8898+a84e86e1.x86_64 fails 2. Testing with qemu level, didn't this this issue. 3. If installing the openvswitch packages(which can provide ovs-vsctl tool), then this issue is gone. But booting vhost-user ports should not reply on ovs. So I think we need to fix it.
Pei, can you attach debug logs please? I'm not sure libvirt can run without ovs-vsctl. Maybe it can be tolerant when the binary is missing on domain startup (but not really [1]), but for everything else (e.g. interface statistics) it needs to use the binary simply because ovs has no library and using the binary is the only way. 1: when starting a domain with a vhostuser <interface/> libvirt queries ovs for the interface name so that it can be reported in the domain XML. So perhaps the only solution is to fix dependencies in .spec file? I thought we have such dependency but I don't see it.
Created attachment 1745475 [details] libvitd log (In reply to Michal Privoznik from comment #1) > Pei, can you attach debug logs please? I'm not sure libvirt can run without Hello Michal, Please see the attachment. > ovs-vsctl. Maybe it can be tolerant when the binary is missing on domain > startup (but not really [1]), but for everything else (e.g. interface > statistics) it needs to use the binary simply because ovs has no library and > using the binary is the only way. > > 1: when starting a domain with a vhostuser <interface/> libvirt queries ovs > for the interface name so that it can be reported in the domain XML. DPDK can also provide the vhost-user interface. So ovs is not a must for this scenario. Thanks. Best regards, Pei > > So perhaps the only solution is to fix dependencies in .spec file? I thought > we have such dependency but I don't see it.
Alright, so dpdk-testpmd is a test utility, isn't it? Because so far, libvirt only supports DPDK via openvswitch. Nothing in the code base suggests we would try work with something else. BTW: That's how OpenStack uses it, isn't it? I can make domain startup ignore this error, but there are other APIs that will still fail (e.g. getting statistics: virsh domifstat). But since this is for testing only (am I correct in assuming that you use dpdk-testpmd only so that you don't have to set up whole OVS machinery?), I guess we can get away with it.
Patch proposed upstream: https://www.redhat.com/archives/libvir-list/2021-January/msg00578.html
(In reply to Michal Privoznik from comment #4) > Alright, so dpdk-testpmd is a test utility, isn't it? Because so far, > libvirt only supports DPDK via openvswitch. Nothing in the code base > suggests we would try work with something else. BTW: That's how OpenStack > uses it, isn't it? I can make domain startup ignore this error, but there > are other APIs that will still fail (e.g. getting statistics: virsh > domifstat). But since this is for testing only (am I correct in assuming > that you use dpdk-testpmd only so that you don't have to set up whole OVS > machinery?), I guess we can get away with it. Hello Michal, You are right, dpdk-testpmd is a test utility, and OpenStack only works with ovs-dpdk. But users may use dpdk + vhost-user. Below doc comes from dpdk official website, it shows some usage example with dpdk + vhost-user. https://doc.dpdk.org/guides/nics/vhost.html Thanks for your patch. Best regards, Pei
Merged upstream as: 6f06ae15d0 openvswitch: Check if OVS_VSCTL exists when getting interface name v7.0.0-rc1-5-g6f06ae15d0
Verify this bug with libvirt-daemon-7.0.0-1.module+el8.4.0+9464+3e71831a.x86_64: S1: Start guest with dpdk-testpmd created vhost-user interface 1. prepare guest which have vhost-user interface like this: <interface type='vhostuser'> <mac address='88:66:da:5f:dd:12'/> <source type='unix' path='/tmp/vhost-user1' mode='server'/> <model type='virtio'/> <driver name='vhost' queues='2' rx_queue_size='1024' iommu='on' ats='on'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </interface> <interface type='vhostuser'> <mac address='88:66:da:5f:dd:13'/> <source type='unix' path='/tmp/vhost-user2' mode='server'/> <model type='virtio'/> <driver name='vhost' queues='2' rx_queue_size='1024' iommu='on' ats='on'/> <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/> </interface> 2. open a new terminal to run dpdk-testpmd # dpdk-testpmd -l 1-3 --socket-mem 1024,1024 -n 4 --vdev 'net_vhost0,iface=/tmp/vhost-user1,queues=2,client=1,iommu-support=1' --vdev 'net_vhost1,iface=/tmp/vhost-user2,queues=2,client=1,iommu-support=1' -d /usr/lib64/librte_net_vhost.so.21 -- --portmask=f -i --rxd=512 --txd=512 --rxq=2 --txq=2 --nb-cores=2 --forward-mode=io 3. start guest # virsh start vm1 Domain 'vm1' started 4. check guest xml and there is no target element for vhostuser interface: # virsh dumpxml vm1 ... <interface type='vhostuser'> <mac address='88:66:da:5f:dd:12'/> <source type='unix' path='/tmp/vhost-user1' mode='server'/> <model type='virtio'/> <driver name='vhost' queues='2' rx_queue_size='1024' iommu='on' ats='on'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </interface> <interface type='vhostuser'> <mac address='88:66:da:5f:dd:13'/> <source type='unix' path='/tmp/vhost-user2' mode='server'/> <model type='virtio'/> <driver name='vhost' queues='2' rx_queue_size='1024' iommu='on' ats='on'/> <alias name='net1'/> <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/> </interface> ... 5. test domstats command run without error: # virsh domstats Domain: 'vm1' ... net.count=2 block.count=0
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (virt:av bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:2098