Bug 1291851

Please note that virtio-vsock is now available in upstream Linux and QEMU.  If you have any questions about the feature, please let me know so we can discuss it.

Basic "getting started" information is available here: http://qemu-project.org/Features/VirtioVsock

There is now a copr repository so you can install the kernel, qemu-kvm, and nc-vsock utility on Fedora 24:
https://copr.fedorainfracloud.org/coprs/stefanha/vsock/

Note that the qemu-ga (guest agent) in these RPMs has AF_VSOCK support.  Use qemu-ga -m vsock-listen -p 3:1234 to listen on port 1234 (assuming the guest CID is 3 on QEMU's command-line -device vhost-vsock-pci,guest-cid=3 option).

Hi Ján,
I suggest the following CID allocation approach:

libvirtd instances manage a CID address range from which CIDs can be allocated automatically for guests.  This is similar to DHCP ranges and it partitions the address space, allowing some ranges to be statically assigned or even owned by other libvirtd sessions without collisions.

Users should also be able to statically assign CIDs to guests.  This is useful if a user wants a persistent CID for a specific guest.

The driver interface works as follows:

  vhostfd = open("/dev/vhost-vsock")
  uint64_t guest_cid = ...;
  ioctl(vhostfd, VHOST_VSOCK_SET_GUEST_CID, &guest_cid);

The ioctl fails with EADDRINUSE if another vhost-vsock instance already has the CID assigned.  The ioctl is idempotent and succeeds if you assign the same CID again.  It fails with EINVAL if the CID is invalid (<2 or >=UINT32_MAX).

Once the CID has been set it's time to launch QEMU:

  -device vhost-vsock-pci,vhostfd=<fd>,guest-cid=<guest-cid>

(Or using device_add hotplug.)

Note that it's necessary to pass the guest CID to QEMU, but that should be easy because libvirt already knows it.

Linux 4.13 and later have a static device number assigned to /dev/vhost-vsock so that the kernel auto-loads vhost_vsock.ko when the device node is opened for the first time.  Older Linux kernels require an explicit modprobe vhost-vsock (yes, just like /dev/vhost-net used to!).

Hope this is useful info for starting the libvirt work.  Please let me know if you have questions!

Initial version of upstream patches:
https://www.redhat.com/archives/libvir-list/2018-May/msg01517.html

v2:
https://www.redhat.com/archives/libvir-list/2018-May/msg01757.html

Pushed upstream as of:
commit b8b42ca036adbfaac1741c8efe389cd1403e220b
Author:     Ján Tomko <jtomko>
AuthorDate: 2018-05-22 15:57:47 +0200
Commit:     Ján Tomko <jtomko>
CommitDate: 2018-05-29 15:42:04 +0200

    qemu: add support for vhost-vsock-pci
    
    Create a new vsock endpoint by opening /dev/vhost-vsock,
    set the requested CID via ioctl (or assign a free one if auto='yes'),
    pass the file descriptor to QEMU and build the command line.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1291851
    Signed-off-by: Ján Tomko <jtomko>

git describe: v4.3.0-372-gb8b42ca036

Filed a bug against selinux-policy:
https://bugzilla.redhat.com/show_bug.cgi?id=1584011

Follow-up series implementing hotplug:
https://www.redhat.com/archives/libvir-list/2018-May/msg02239.html

And another series changing the element name from <source cid=''> to <cid address=''>:
https://www.redhat.com/archives/libvir-list/2018-June/msg00037.html

The rename is now pushed:
commit 023ea2a86938a6ecb5323a561f422c4951c8bf39
Author:     Ján Tomko <jtomko>
AuthorDate: 2018-06-01 13:22:56 +0200
Commit:     Ján Tomko <jtomko>
CommitDate: 2018-06-01 14:31:19 +0200

    conf: rename <vsock><source> to <vsock><cid>
    
    To avoid the <source> vs. <target> confusion,
    change <source auto='no' cid='3'/> to:
    <cid auto='no' address='3'/>
    
    Signed-off-by: Ján Tomko <jtomko>
    Suggested-by: Daniel P. Berrangé <berrange>
    Acked-by: Peter Krempa <pkrempa>
    Reviewed-by: Daniel P. Berrangé <berrange>

git describe: v4.4.0-rc1-12-g023ea2a869

There was one more follow-up change before the release:
commit 8a7003f66944721ec391e13e65bbc5fdfdec3cea
Author:     Ján Tomko <jtomko>
CommitDate: 2018-06-04 21:42:40 +0200

    qemu: check for QEMU_CAPS_DEVICE_VHOST_VSOCK
    
    My commit b8b42ca added support for formatting the vsock
    command line without actually checking if it's supported.
    
    Add it to the per-device validation function.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1291851
    
    Reported-by: John Ferlan <jferlan>
    Signed-off-by: Ján Tomko <jtomko>
    Reviewed-by: Jiri Denemark <jdenemar>

git describe: v4.4.0-rc2-4-g8a7003f669 contains: v4.4.0~1

Backport of the hotplug series:
http://post-office.corp.redhat.com/archives/rhvirt-patches/2018-June/msg00055.html

Hi, Ján,

It reports error when i try to hotplug/coldplug vsock device to the guest with libvirt-4.4.0-2.el7.x86_64. Could you help to check whether the patch http://post-office.corp.redhat.com/archives/rhvirt-patches/2018-June/msg00055.html includes in the libvirt-4.4.0-2.el7.x86_64 or not please?
Thanks a lot.

Test steps:
1.Start a guest:
#virsh start iommu1

2.Prepare vsock device xml:
#cat vsock.xml
<vsock model='virtio'>
   <cid auto='yes'/>
</vsock>

3.Hotplug the vsock device to the guest:
#virsh attach-device iommu1 vsock.xml
error: Failed to attach device from /root/vsock.xml
error: Operation not supported: live attach of device 'vsock' is not supported

4.Coldplug the vsock device to the guest:
#virsh attach-device iommu1 vsock.xml --config
error: Failed to attach device from /root/vsock.xml
error: Operation not supported: persistent attach of device 'vsock' is not supported

Hi,

libvirt-4.4.0-2.el7.x86_64 does not have the additional hotplug patches. Only what was picked up by rebase to 4.4.0.

Sorry for the confusion.

Verified with libvirt-4.5.0-6.el7.x86_64 and qemu-kvm-rhev-2.12.0-10.el7.x86_64.

Test steps:
Scenario 1: Start a guest with vsock device:
1).Add vsock model on host os:
#modprobe vhost_vsock

2).Start a guest with vsock device:
#virsh start rhel7.6
Domain rhel7.6 started

#virsh dumpxml rhel7.6 | grep -A5 vsock
# virsh dumpxml rhel7.6 | grep -A5 vsock
    <vsock model='virtio'>
      <cid auto='no' address='3'/>
      <alias name='ua-04c3388d-4e33-4023-84de-a2205c777asdfdsf'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
    </vsock>

3)Check the qemu cmd line:
#ps aux | grep -i cid:
-device vhost-vsock-pci,id=ua-04c2decd-4e33-4023-84de-a2205sdfsfdfdsf,guest-cid=3,vhostfd=24,bus=pci.8,addr=0x0

4).Download nc-vosck.c both on guest and host:
#git clone https://github.com/stefanha/nc-vsock.git

5).Start listening socket inside guest:
(guest)#./nc-vsock -l 1234

6).Connect guest cid from host and input some chars:
(host)#./nc-vsock 3 1234
Test for vsock device

7).Should see the same chars input in step 6 in the terminal started in the step 4;

8).Start a guest with auto address cid, and repeat step4-7, it also works well;



Scenario 2: Start two guest with the same cid and auto='no':
1)Start a guest with cid=3 and auto='no':
#virsh start rhel7.6
Domain rhel7.6 started

#virsh dumpxml rhel7.6 | grep -A5 vsock
# virsh dumpxml rhel7.6 | grep -A5 vsock
    <vsock model='virtio'>
      <cid auto='no' address='3'/>
      <alias name='ua-04c3388d-4e33-4023-84de-a2205c777asdfdsf'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
    </vsock>

2)Start another guest with cid=3 and auto='no':
#virsh edit vm1:
<vsock model='virtio'>
      <cid auto='no' address='3'/>
      <alias name='ua-04c3388d-4e33-4023-84de-a2205c777asdfdsf'/>
</vsock>

#virsh start vm1:
error: failed to set guest cid: Address already in use


Scenario 3: Hotplug/hotunplug vsock device:
1)Prepare a vsock device xml:
#cat vsock.xml
# cat /xml/vsock.xml
<vsock model='virtio'>
      <cid auto='no' address='999'/>
   <alias name='ua-04csdfasfdcd-4e33-4023-84de-a2205c777asdfdsf'/>
</vsock>

2)Hotplug the vsock device to a runnig guest:
#virsh attach-device iommu1 /xml/vsock.xml
Device attached successfully

3)Check the live xml:
#virsh dumpxml iommu1
<vsock model='virtio'>
      <cid auto='no' address='999'/>
      <alias name='ua-04c2decd-4e33-4023-84de-a2205sdfsfdfdsf'/>
      <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
</vsock>

4.Check the vsock device in the guest os:
(guest)#lspci
04:00.0 Communication controller: Red Hat, Inc. Device 1053 (rev 01)

5.Hotunplug the vsock device:
#virsh detach-device iommu1 /xml/vsock.xml

6.Check the live xml and no vsock device;

7.Check the devices in the guest os, no vsock device found;

8.Repeat step 2-7 with auto='yes', it also works well.


Also test coldplug/coldunplug vsock device, try to start guest with two vsock devices, edit vsock device with invalid cid, do migration with vsock device, start guest without vhost_vsock module in the host os, all the results are as expected.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:3113