Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1916832

Summary: [DF] [Upgrades] Handle a to-be-introduced OVS/DPDK flag to facilitate upgrades from OSP-13 to OSP-16
Product: Red Hat OpenStack Reporter: David Vallee Delisle <dvd>
Component: openstack-tripleo-heat-templatesAssignee: Jaganathan Palanisamy <jpalanis>
Status: CLOSED WONTFIX QA Contact: Joe H. Rahme <jhakimra>
Severity: high Docs Contact:
Priority: high    
Version: 16.1 (Train)CC: hakhande, kchamart, kmehta, maxime.coquelin, mburns, smooney
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-01-27 13:54:36 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1917817    
Bug Blocks: 1902631, 2244628    

Description David Vallee Delisle 2021-01-15 16:12:30 UTC 
This bug was initially created as a copy of Bug #1902631

I am copying this bug because: 

OVS will probably end up introducing a new global flag in the OVSDB, OpenvSwitch table that mimics the legacy behaviour
TripleO will need to have a config parameter to set that flag
This flag should be disabled for new deployments and enabled during upgrades (disabled afterward).

Description of problem:

Live-migration of DPDK VMs from compute node 13 to 16.1 fails. Cold migration works fine.

(overcloud) [stack@undercloud ~]$ openstack server list --long | grep dpdk-inst2
| 1102df92-4228-4ebe-855a-02fe4b0fec96 | dpdk-inst2 | ACTIVE | None       | Running     | dpdk=192.168.24.250 | rhel7      | ca78bb6a-0630-42ac-9b7c-e52f5d4e81a5 |             |           | nova              | overcloud-computeovsdpdk-0.localdomain | 


The instance does end up on the destination compute node(16.1) but it seems to have failed due to unsupported features. 
~~~
[root@overcloud-computeovsdpdk-1 ~]# tail -n8  /var/log/libvirt/qemu/instance-0000000e.log
2020-11-27T08:11:40.854290Z qemu-kvm: -chardev socket,id=charnet0,path=/var/lib/vhost_sockets/vhu52667eae-a7,server: info: QEMU waiting for connection on: disconnected:unix:/var/lib/vhost_sockets/vhu52667eae-a7,server
char device redirected to /dev/pts/2 (label charserial0)
2020-11-27T08:11:42.158333Z qemu-kvm: -device cirrus-vga,id=video0,bus=pci.0,addr=0x2: warning: 'cirrus-vga' is deprecated, please use a different VGA card instead
2020-11-27T08:11:49.771254Z qemu-kvm: Features 0x130afe7a2 unsupported. Allowed features: 0x178bfa7e6
2020-11-27T08:11:49.771301Z qemu-kvm: Failed to load virtio-net:virtio
2020-11-27T08:11:49.771312Z qemu-kvm: error while loading state for instance 0x0 of device '0000:00:03.0/virtio-net'
2020-11-27T08:11:49.771683Z qemu-kvm: load of migration failed: Operation not permitted
2020-11-27 08:11:49.987+0000: shutting down, reason=failed
~~~


nova-compute debug logs on the source node
~~~
2020-11-30 06:31:40.245 7 DEBUG nova.virt.libvirt.migration [-] [instance: 1102df92-4228-4ebe-855a-02fe4b0fec96] Current 50 elapsed 9 steps [(0, 50), (300, 95), (600, 140), (900, 185), (1200, 230), (1500, 275), (1800, 320), (2100, 365), (2400, 410), (270
0, 455), (3000, 500)] update_downtime /usr/lib/python3.6/site-packages/nova/virt/libvirt/migration.py:501
2020-11-30 06:31:40.246 7 DEBUG nova.virt.libvirt.migration [-] [instance: 1102df92-4228-4ebe-855a-02fe4b0fec96] Downtime does not need to change update_downtime /usr/lib/python3.6/site-packages/nova/virt/libvirt/migration.py:513
2020-11-30 06:31:40.256 7 ERROR nova.virt.libvirt.driver [-] [instance: 1102df92-4228-4ebe-855a-02fe4b0fec96] Live Migration failure: internal error: qemu unexpectedly closed the monitor: 2020-11-30T06:31:31.524866Z qemu-kvm: -chardev socket,id=charnet0,
path=/var/lib/vhost_sockets/vhu52667eae-a7,server: info: QEMU waiting for connection on: disconnected:unix:/var/lib/vhost_sockets/vhu52667eae-a7,server
2020-11-30T06:31:32.395046Z qemu-kvm: -device cirrus-vga,id=video0,bus=pci.0,addr=0x2: warning: 'cirrus-vga' is deprecated, please use a different VGA card instead
2020-11-30T06:31:39.931517Z qemu-kvm: Features 0x130afe7a2 unsupported. Allowed features: 0x178bfa7e6
2020-11-30T06:31:39.931562Z qemu-kvm: Failed to load virtio-net:virtio
2020-11-30T06:31:39.931573Z qemu-kvm: error while loading state for instance 0x0 of device '0000:00:03.0/virtio-net'
2020-11-30T06:31:39.931948Z qemu-kvm: load of migration failed: Operation not permitted: libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2020-11-30T06:31:31.524866Z qemu-kvm: -chardev socket,id=charnet0,path=/var/lib/vhost_soc
kets/vhu52667eae-a7,server: info: QEMU waiting for connection on: disconnected:unix:/var/lib/vhost_sockets/vhu52667eae-a7,server
2020-11-30 06:31:40.257 7 DEBUG nova.virt.libvirt.driver [-] [instance: 1102df92-4228-4ebe-855a-02fe4b0fec96] Migration operation thread notification thread_finished /usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py:9144
2020-11-30 06:31:40.281 7 INFO nova.compute.manager [req-afa6db20-2e13-4fcf-bed6-606de5f0fc36 - - - - -] [instance: 1102df92-4228-4ebe-855a-02fe4b0fec96] During sync_power_state the instance has a pending task (migrating). Skip.
2020-11-30 06:31:40.749 7 DEBUG nova.virt.libvirt.migration [-] [instance: 1102df92-4228-4ebe-855a-02fe4b0fec96] VM running on src, migration failed _log /usr/lib/python3.6/site-packages/nova/virt/libvirt/migration.py:419
2020-11-30 06:31:40.750 7 DEBUG nova.virt.libvirt.driver [-] [instance: 1102df92-4228-4ebe-855a-02fe4b0fec96] Fixed incorrect job type to be 4 _live_migration_monitor /usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py:8958
2020-11-30 06:31:40.750 7 ERROR nova.virt.libvirt.driver [-] [instance: 1102df92-4228-4ebe-855a-02fe4b0fec96] Migration operation has aborted
~~~


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
While upgrading osp13 compute node, migrate the workload to osp16.1 compute node

Actual results:
Workload migration fails, which prevents upgrading of the compute node to 16.1

Expected results:
Workload should migrate to osp16.1 compute node to perform upgrade

Additional info:
Comment 7 smooney 2021-01-27 13:54:36 UTC 
closeing this as wont fix as we decied not to add a new ovs parmater there for there is nothing to do.

the previous bug was closed as a duplciat of bz#1916869 which is a documentaion update to note that cold migration must be used