1834592 – [OSP 16.1] Fail to spawn guest instance with SR-IOV VF on Intel X710 NIC

Bug 1834592 - [OSP 16.1] Fail to spawn guest instance with SR-IOV VF on Intel X710 NIC

Summary: [OSP 16.1] Fail to spawn guest instance with SR-IOV VF on Intel X710 NIC

Keywords:
Status:	CLOSED DUPLICATE of bug 1828834
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-neutron
Sub Component:
Version:	16.1 (Train)
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	high
Target Milestone:	---
Target Release:	---
Assignee:	Assaf Muller
QA Contact:	Eran Kuris
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2020-05-12 03:15 UTC by Vadim Khitrin
Modified:	2020-05-12 10:26 UTC (History)
CC List:	4 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2020-05-12 10:26:10 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Description Vadim Khitrin 2020-05-12 03:15:11 UTC

Description of problem:
When attempting to spawn a guest instance with SR-IOV VF from an Intel X710 NIC, the spawn fails.

It is important to note that an instance is spawned correctly when using SR-IOV PF.

Compute hosting instance:
computeovsdpdksriov-1

SR-IOV VF port:
e5af98ef-cd0e-478f-b88e-e8da68272245

instance ID:
963f96c4-5641-42e3-8a0a-b1493a0809f3

QEMU instance name:
instance-00000059

The following error observed in nova-compute.log:
nova-compute.log:2020-05-11 14:33:37.923 7 ERROR nova.compute.manager [instance: 963f96c4-5641-42e3-8a0a-b1493a0809f3] Traceback (most recent call last):
nova-compute.log:2020-05-11 14:33:37.923 7 ERROR nova.compute.manager [instance: 963f96c4-5641-42e3-8a0a-b1493a0809f3]   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 6351, in _create_domain_and_network
nova-compute.log:2020-05-11 14:33:37.923 7 ERROR nova.compute.manager [instance: 963f96c4-5641-42e3-8a0a-b1493a0809f3]     network_info)
nova-compute.log:2020-05-11 14:33:37.923 7 ERROR nova.compute.manager [instance: 963f96c4-5641-42e3-8a0a-b1493a0809f3]   File "/usr/lib64/python3.6/contextlib.py", line 88, in __exit__
nova-compute.log:2020-05-11 14:33:37.923 7 ERROR nova.compute.manager [instance: 963f96c4-5641-42e3-8a0a-b1493a0809f3]     next(self.gen)
nova-compute.log:2020-05-11 14:33:37.923 7 ERROR nova.compute.manager [instance: 963f96c4-5641-42e3-8a0a-b1493a0809f3]   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 478, in wait_for_instance_event
nova-compute.log:2020-05-11 14:33:37.923 7 ERROR nova.compute.manager [instance: 963f96c4-5641-42e3-8a0a-b1493a0809f3]     actual_event = event.wait()
nova-compute.log:2020-05-11 14:33:37.923 7 ERROR nova.compute.manager [instance: 963f96c4-5641-42e3-8a0a-b1493a0809f3]   File "/usr/lib/python3.6/site-packages/eventlet/event.py", line 125, in wait
nova-compute.log:2020-05-11 14:33:37.923 7 ERROR nova.compute.manager [instance: 963f96c4-5641-42e3-8a0a-b1493a0809f3]     result = hub.switch()
nova-compute.log:2020-05-11 14:33:37.923 7 ERROR nova.compute.manager [instance: 963f96c4-5641-42e3-8a0a-b1493a0809f3]   File "/usr/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 298, in switch
nova-compute.log:2020-05-11 14:33:37.923 7 ERROR nova.compute.manager [instance: 963f96c4-5641-42e3-8a0a-b1493a0809f3]     return self.greenlet.switch()
nova-compute.log:2020-05-11 14:33:37.923 7 ERROR nova.compute.manager [instance: 963f96c4-5641-42e3-8a0a-b1493a0809f3] eventlet.timeout.Timeout: 300 seconds
nova-compute.log:2020-05-11 14:33:37.923 7 ERROR nova.compute.manager [instance: 963f96c4-5641-42e3-8a0a-b1493a0809f3]
nova-compute.log:2020-05-11 14:33:37.923 7 ERROR nova.compute.manager [instance: 963f96c4-5641-42e3-8a0a-b1493a0809f3] During handling of the above exception, another exception occurred:
nova-compute.log:2020-05-11 14:33:37.923 7 ERROR nova.compute.manager [instance: 963f96c4-5641-42e3-8a0a-b1493a0809f3]
nova-compute.log:2020-05-11 14:33:37.923 7 ERROR nova.compute.manager [instance: 963f96c4-5641-42e3-8a0a-b1493a0809f3] Traceback (most recent call last):
nova-compute.log:2020-05-11 14:33:37.923 7 ERROR nova.compute.manager [instance: 963f96c4-5641-42e3-8a0a-b1493a0809f3]   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 2437, in _build_and_run_instance
nova-compute.log:2020-05-11 14:33:37.923 7 ERROR nova.compute.manager [instance: 963f96c4-5641-42e3-8a0a-b1493a0809f3]     block_device_info=block_device_info)
nova-compute.log:2020-05-11 14:33:37.923 7 ERROR nova.compute.manager [instance: 963f96c4-5641-42e3-8a0a-b1493a0809f3]   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 3592, in spawn
nova-compute.log:2020-05-11 14:33:37.923 7 ERROR nova.compute.manager [instance: 963f96c4-5641-42e3-8a0a-b1493a0809f3]     power_on=power_on)
nova-compute.log:2020-05-11 14:33:37.923 7 ERROR nova.compute.manager [instance: 963f96c4-5641-42e3-8a0a-b1493a0809f3]   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 6372, in _create_domain_and_network
nova-compute.log:2020-05-11 14:33:37.923 7 ERROR nova.compute.manager [instance: 963f96c4-5641-42e3-8a0a-b1493a0809f3]     raise exception.VirtualInterfaceCreateException()
nova-compute.log:2020-05-11 14:33:37.923 7 ERROR nova.compute.manager [instance: 963f96c4-5641-42e3-8a0a-b1493a0809f3] nova.exception.VirtualInterfaceCreateException: Virtual Interface creation failed

No other errors regarding to port were found in neutron logs.

QEMU log of the instance:
[root@computeovsdpdksriov-1 nova]# cat /var/log/libvirt/qemu/instance-00000059.log
2020-05-11 14:28:32.785+0000: starting up libvirt version: 6.0.0, package: 17.module+el8.2.0+6257+0d066c28 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2020-04-08-16:41:56, ), qemu version: 4.2.0qemu-kvm-4.2.0-19.module+el8.2.0+6296+6b821950, kernel: 4.18.0-193.el8.x86_64, hostname: computeovsdpdksriov-1.lo
caldomain
LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
HOME=/var/lib/libvirt/qemu/domain-21-instance-00000059 \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-21-instance-00000059/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-21-instance-00000059/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-21-instance-00000059/.config \
QEMU_AUDIO_DRV=none \
/usr/libexec/qemu-kvm \
-name guest=instance-00000059,debug-threads=on \
-S \
-object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-21-instance-00000059/master-key.aes \
-machine pc-i440fx-rhel7.6.0,accel=kvm,usb=off,dump-guest-core=off \
-cpu Broadwell-IBRS,vme=on,ss=on,vmx=on,f16c=on,rdrand=on,hypervisor=on,arat=on,tsc-adjust=on,umip=on,stibp=on,arch-capabilities=on,ssbd=on,xsaveopt=on,pdpe1gb=on,abm=on,ibpb=on,amd-ssbd=on,skip-l1dfl-vmentry=on \
-m 8192 \
-overcommit mem-lock=off \
-smp 6,sockets=3,dies=1,cores=1,threads=2 \
-object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu/21-instance-00000059,share=yes,size=8589934592,host-nodes=0,policy=bind \
-numa node,nodeid=0,cpus=0-5,memdev=ram-node0 \
-uuid 963f96c4-5641-42e3-8a0a-b1493a0809f3 \
-smbios 'type=1,manufacturer=Red Hat,product=OpenStack Compute,version=20.2.1-0.20200424133447.118ee68.el8ost,serial=963f96c4-5641-42e3-8a0a-b1493a0809f3,uuid=963f96c4-5641-42e3-8a0a-b1493a0809f3,family=Virtual Machine' \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=33,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-shutdown \
-boot strict=on \
-device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 \
-blockdev '{"driver":"file","filename":"/var/lib/nova/instances/_base/7423ddeaaf92eb6a7c8a2c1b9e9903d7bb5dfae4","node-name":"libvirt-3-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-3-format","read-only":true,"cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-3-storage"}' \
-blockdev '{"driver":"file","filename":"/var/lib/nova/instances/963f96c4-5641-42e3-8a0a-b1493a0809f3/disk","node-name":"libvirt-2-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-2-storage","backing":"libvirt-3-format"}' \
-device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=libvirt-2-format,id=virtio-disk0,bootindex=1,write-cache=on \
-blockdev '{"driver":"file","filename":"/var/lib/nova/instances/963f96c4-5641-42e3-8a0a-b1493a0809f3/disk.config","node-name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":true,"cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-1-storage"}' \
-device ide-cd,bus=ide.0,unit=0,drive=libvirt-1-format,id=ide0-0-0,write-cache=on \
-chardev socket,id=charnet0,path=/var/lib/vhost_sockets/vhufdd66ca4-c5,server \
-netdev vhost-user,chardev=charnet0,id=hostnet0 \
-device virtio-net-pci,rx_queue_size=1024,tx_queue_size=1024,netdev=hostnet0,id=net0,mac=fa:16:3e:ac:8f:7c,bus=pci.0,addr=0x3 \
-add-fd set=1,fd=36 \
-chardev pty,id=charserial0,logfile=/dev/fdset/1,logappend=on \
-device isa-serial,chardev=charserial0,id=serial0 \
-device usb-tablet,id=input0,bus=usb.0,port=1 \
-vnc 10.10.130.195:0 \
-device cirrus-vga,id=video0,bus=pci.0,addr=0x2 \
-device vfio-pci,host=0000:06:0a.3,id=hostdev0,bus=pci.0,addr=0x5 \
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
2020-05-11T14:28:32.836394Z qemu-kvm: -chardev socket,id=charnet0,path=/var/lib/vhost_sockets/vhufdd66ca4-c5,server: info: QEMU waiting for connection on: disconnected:unix:/var/lib/vhost_sockets/vhufdd66ca4-c5,server
char device redirected to /dev/pts/1 (label charserial0)
2020-05-11T14:28:35.615580Z qemu-kvm: -device cirrus-vga,id=video0,bus=pci.0,addr=0x2: warning: 'cirrus-vga' is deprecated, please use a different VGA card instead
2020-05-11T14:33:36.458277Z qemu-kvm: terminating on signal 15 from pid 23995 (/usr/sbin/libvirtd)
2020-05-11 14:33:36.658+0000: shutting down, reason=destroyed


Version-Release number of selected component (if applicable):
compose: RHOS-16.1-RHEL-8-20200505.n.0

python3-neutron-15.0.3-0.20200505163434.b019821.el8ost.noarch
openstack-neutron-common-15.0.3-0.20200505163434.b019821.el8ost.noarch
python3-neutron-dynamic-routing-15.0.1-0.20200311020838.c2cf1cd.el8ost.noarch
openstack-neutron-ml2-15.0.3-0.20200505163434.b019821.el8ost.noarch
python3-neutronclient-6.14.0-0.20200310192910.115f60f.el8ost.noarch
puppet-neutron-15.5.1-0.20200430113430.731a8fd.el8ost.noarch
python3-neutron-lib-1.29.1-0.20200310214054.4ef4b71.el8ost.noarch
openstack-neutron-15.0.3-0.20200505163434.b019821.el8ost.noarch
openstack-neutron-sriov-nic-agent-15.0.3-0.20200505163434.b019821.el8ost.noarch
qemu-kvm-block-ssh-4.2.0-19.module+el8.2.0+6296+6b821950.x86_64
qemu-kvm-4.2.0-19.module+el8.2.0+6296+6b821950.x86_64
qemu-kvm-block-iscsi-4.2.0-19.module+el8.2.0+6296+6b821950.x86_64
qemu-img-4.2.0-19.module+el8.2.0+6296+6b821950.x86_64
ipxe-roms-qemu-20181214-5.git133f4c47.el8.noarch
qemu-kvm-common-4.2.0-19.module+el8.2.0+6296+6b821950.x86_64
qemu-kvm-block-curl-4.2.0-19.module+el8.2.0+6296+6b821950.x86_64
qemu-kvm-block-gluster-4.2.0-19.module+el8.2.0+6296+6b821950.x86_64
qemu-kvm-core-4.2.0-19.module+el8.2.0+6296+6b821950.x86_64
qemu-kvm-block-rbd-4.2.0-19.module+el8.2.0+6296+6b821950.x86_64
libvirt-daemon-driver-secret-6.0.0-17.module+el8.2.0+6257+0d066c28.x86_64
libvirt-daemon-driver-storage-mpath-6.0.0-17.module+el8.2.0+6257+0d066c28.x86_64
libvirt-daemon-driver-storage-core-6.0.0-17.module+el8.2.0+6257+0d066c28.x86_64
libvirt-daemon-driver-storage-disk-6.0.0-17.module+el8.2.0+6257+0d066c28.x86_64
libvirt-daemon-kvm-6.0.0-17.module+el8.2.0+6257+0d066c28.x86_64
libvirt-bash-completion-6.0.0-17.module+el8.2.0+6257+0d066c28.x86_64
python3-libvirt-6.0.0-1.module+el8.2.0+5453+31b2b136.x86_64
libvirt-daemon-driver-nodedev-6.0.0-17.module+el8.2.0+6257+0d066c28.x86_64
libvirt-daemon-driver-storage-rbd-6.0.0-17.module+el8.2.0+6257+0d066c28.x86_64
libvirt-daemon-driver-storage-logical-6.0.0-17.module+el8.2.0+6257+0d066c28.x86_64
libvirt-daemon-driver-storage-6.0.0-17.module+el8.2.0+6257+0d066c28.x86_64
libvirt-client-6.0.0-17.module+el8.2.0+6257+0d066c28.x86_64
libvirt-daemon-driver-nwfilter-6.0.0-17.module+el8.2.0+6257+0d066c28.x86_64
libvirt-daemon-driver-storage-gluster-6.0.0-17.module+el8.2.0+6257+0d066c28.x86_64
libvirt-daemon-driver-storage-scsi-6.0.0-17.module+el8.2.0+6257+0d066c28.x86_64
libvirt-daemon-driver-network-6.0.0-17.module+el8.2.0+6257+0d066c28.x86_64
libvirt-daemon-6.0.0-17.module+el8.2.0+6257+0d066c28.x86_64
libvirt-daemon-driver-storage-iscsi-6.0.0-17.module+el8.2.0+6257+0d066c28.x86_64
libvirt-daemon-driver-interface-6.0.0-17.module+el8.2.0+6257+0d066c28.x86_64
libvirt-libs-6.0.0-17.module+el8.2.0+6257+0d066c28.x86_64
libvirt-daemon-driver-storage-iscsi-direct-6.0.0-17.module+el8.2.0+6257+0d066c28.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Deploy SR-IOV capable deployment
2. Spawn an instance with SR-IOV VF attached to it

Actual results:
Instance fails to spawn.

Expected results:
Instance successfully spawned.

Additional info:
Will attach SOS report in comment.

Comment 1 Vadim Khitrin 2020-05-12 03:15:50 UTC

Forgot to mention that is is an OVS based setup.

Note You need to log in before you can comment on or make changes to this bug.