Bug 1156267

Summary: qemu-kvm process hung when starting a guest with vhostuser interface
Product: Red Hat Enterprise Linux 7 Reporter: Hu Jianwei <jiahu>
Component: qemu-kvm-rhevAssignee: Michael S. Tsirkin <mst>
Status: CLOSED NOTABUG QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 7.1CC: ailan, drjones, dyuan, hhuang, honzhang, huding, juzhang, knoel, mdeng, mst, mzhan, pezhang, qiguo, virt-maint, xfu, xiywang
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-09-09 11:53:25 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Hu Jianwei 2014-10-24 03:48:30 UTC
Description
qemu-kvm process hung when starting a guest with vhostuser interface

Version:
libvirt-1.2.8-5.el7.x86_64
qemu-kvm-rhev-2.1.2-3.el7.x86_64
qemu-kvm-1.5.3-75.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
[root@localhost ~]# virsh dumpxml r7 | grep "/interface" -B6
    <interface type='vhostuser'>
      <mac address='52:54:00:3b:83:1a'/>
      <source type='unix' path='/var/lib/libvirt/qemu/vhost.sock' mode='server'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

Actual results:
As above shown steps, the qemu-kvm process could not boot up normally

Expected results:
The guest can start successfully

Additional info:
When I run a domain with client vhostuser, the first domain can continue to start, maybe my question should be changed to "whether or not qemu process need wait for client connection in the background"

+++++++++++++
file:///usr/share/doc/libvirt-docs-1.2.8/html/formatdomain.html
Since 1.2.7 the vhost-user enables the communication between a QEMU virtual machine and other userspace process using the Virtio transport protocol.
+++++++++++++

the second qemu command:
 /usr/libexec/qemu-kvm -name r7 -S -machine pc-i440fx-rhel7.1.0,accel=kvm,usb=off -m 1024 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid de5cfdda-30b9-4c4f-ab65-e8acd276aab6 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/r7.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -drive file=/var/lib/libvirt/images/r7_1.img,if=none,id=drive-virtio-disk0,format=raw -device virtio-blk-pci,scsi=off,bus=p!
 ci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -chardev socket,id=charnet0,path=/var/lib/libvirt/qemu/vhost.sock -netdev type=vhost-user,id=hostnet0,chardev=charnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:f8:48:85,bus=pci.0,addr=0x8 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -spice port=5903,addr=127.0.0.1,disable-ticketing,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,bus=pci.0,addr=0x2 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on

Log from qemu-kvm:
strace /usr/libexec/qemu-kvm -name r7 -S -machine pc-i440fx-rhel7.1.0,accel=kvm,usb=off -cpu SandyBridge -m 1024 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid de5cfdda-30b9-4c4f-ab65-e8acd276aab6 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/r7.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -drive file=/var/lib/libvirt/images/r7.img,if=none,id=drive-ide0-0-0,format=raw -device ide-hd,bus=i!
 de.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -chardev socket,id=charnet0,path=/var/lib/libvirt/qemu/vhost.sock,server -netdev type=vhost-user,id=hostnet0,chardev=charnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:3b:83:1a,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -spice port=5900,addr=127.0.0.1,disable-ticketing,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,bus=pci.0,addr=0x2 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1 -chardev spicevmc,id=charredir2,name=usbredi!
 r -device usb-redir,chardev=charredir2,id=redir2 -chardev spicevmc,id=
charredir3,name=usbredir -device usb-redir,chardev=charredir3,id=redir3 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on

...
bind(8, {sa_family=AF_LOCAL, sun_path="/var/lib/libvirt/qemu/vhost.sock"}, 110) = 0
listen(8, 1)                            = 0
getsockname(8, {sa_family=AF_LOCAL, sun_path="/var/lib/libvirt/qemu/vhost.sock"}, [35]) = 0
fstat(8, {st_mode=S_IFSOCK|0777, st_size=0, ...}) = 0
fcntl(8, F_GETFL)                       = 0x2 (flags O_RDWR)
write(6, "\1\0\0\0\0\0\0\0", 8)         = 8
write(2, "QEMU waiting for connection on: "..., 77QEMU waiting for connection on: unix:/var/lib/libvirt/qemu/vhost.sock,server
) = 77
accept4(8,
^CProcess 2689 detached
 <detached ...>

If I added nowait option to qemu-kvm command, device 'vhost-user' could not be initialized.

[root@localhost ~]# strace /usr/libexec/qemu-kvm -name r7 -S -machine pc-i440fx-rhel7.1.0,accel=kvm,usb=off -m 1024 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid de5cfdda-30b9-4c4f-ab65-e8acd276aab6 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/r7.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -drive file=/var/lib/libvirt/images/r7_1.img,if=none,id=drive-virtio-disk0,format=raw -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -chardev socket,id=charnet0,path=/var/lib/libvirt/qemu/vhost.sock,server,nowait -netdev type=vhost-user,id=hostnet0,chardev=charnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:3b:83:1a,bus=pci.0,addr=0x8 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -spice port=5905,addr=127.0.0.1,disable-ticketing,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,bus=pci.0,addr=0x2 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on
...
write(2, "2014-10-24T03:44:21.053303Z ", 282014-10-24T03:44:21.053303Z ) = 28
write(2, "qemu-kvm:", 9qemu-kvm:)                = 9
write(2, " -netdev", 8 -netdev)                 = 8
write(2, " type=vhost-user,id=hostnet0,cha"..., 45 type=vhost-user,id=hostnet0,chardev=charnet0) = 45
write(2, ": ", 2: )                       = 2
write(2, "vhost-user does not support a ch"..., 76vhost-user does not support a chardev with the following option:
 wait = off) = 76
write(2, "\n", 1
)                       = 1
write(2, "2014-10-24T03:44:21.053805Z ", 282014-10-24T03:44:21.053805Z ) = 28
write(2, "qemu-kvm:", 9qemu-kvm:)                = 9
write(2, " -netdev", 8 -netdev)                 = 8
write(2, " type=vhost-user,id=hostnet0,cha"..., 45 type=vhost-user,id=hostnet0,chardev=charnet0) = 45
write(2, ": ", 2: )                       = 2
write(2, "No suitable chardev found", 25No suitable chardev found) = 25
write(2, "\n", 1
)                       = 1
write(2, "2014-10-24T03:44:21.054006Z ", 282014-10-24T03:44:21.054006Z ) = 28
write(2, "qemu-kvm:", 9qemu-kvm:)                = 9
write(2, " -netdev", 8 -netdev)                 = 8
write(2, " type=vhost-user,id=hostnet0,cha"..., 45 type=vhost-user,id=hostnet0,chardev=charnet0) = 45
write(2, ": ", 2: )                       = 2
write(2, "Device 'vhost-user' could not be"..., 44Device 'vhost-user' could not be initialized) = 44
write(2, "\n", 1
)                       = 1
unlink("/dev/shm/spice.16619")          = 0
close(3)                                = 0
exit_group(1)                           = ?
+++ exited with 1 +++

Comment 2 Michael S. Tsirkin 2014-11-02 08:41:55 UTC
I think it's not a bug.
If you run as a server, you are waiting for client to connect.
Does guest run once you connect a client?

Comment 3 Hu Jianwei 2014-11-03 02:26:34 UTC
I don't think so, a NIC device is a optional device for domain, not like cpu, memory and pci-root devices(they are mandatory), the qemu process should not wait such device until client connected.
But, if you insist on not a bug, it's necessary to add clear and accurate usage into docs or user guide.

Comment 4 Hu Jianwei 2014-11-03 04:31:11 UTC
> Does guest run once you connect a client?

Yes, the domain can continue to run.

Comment 7 Michael S. Tsirkin 2015-09-09 11:53:25 UTC
closing, pls feel free to open a documentation bz.