Created attachment 1710509 [details] Virtual Console Hang on BM node Description of problem: With kernel args issue fixed (BZ #1858673), ComputeOvsDpdkSriov node hangs on leapp upgrade and reboot. Node is not recoverable even after rebooting manually (login via ssh and virtual console is not working).
On manually rebooting, "sriov_config" service is failed (observed on the virtual console logs). Trying to disable SR-IOV and perform the upgrade to isolate the issue.
computeovsdpdksriov node does not have interfaces with ethx naming. And the workaround finds all the interfaces with names starting as "eth", so the workaround is not applied in the compute nodes. We can rule out the workarounds causing trouble. I have also removed the "sriov_config" service and data files before the reboot as this service was throwing errors on manual reboot. But still the error is present. Node hangs on reboot at the same place. I have tried to login via root using the virtual console, but it is throwing "root: no shell: Permission denied". I am working with Sanjay to create DPDK only environments parallelly.
Able to reproduced in stanadlone RHEL 7.8 to 8.2 upgade with only sriov VFs enabled. Leapp upgrade hangs as like the compute nodes. Looking at the errors, it looks like leapp is storing all the interfaces details in the "/var/lib/leapp/leapp.db" file on RHEL 7.8 machine, which has VFs devices also. Once upgraded to 8.2, leapp compares RHEL 7.8 interfaces from DB with the found interfaces. It is not able to find the VFs, because of which an exception is throwin in the persistentnetnamesconfig actror. https://github.com/oamg/leapp-repository/blob/v0.10.0/repos/system_upgrade/el7toel8/actors/persistentnetnamesconfig/actor.py#L50 ~~~~~ def process(self): rhel7_ifaces = next(self.consume(PersistentNetNamesFacts)).interfaces rhel8_ifaces = next(self.consume(PersistentNetNamesFactsInitramfs)).interfaces >>> Exception occurs here rhel7_ifaces_map = {iface.mac: iface for iface in rhel7_ifaces} rhel8_ifaces_map = {iface.mac: iface for iface in rhel8_ifaces} ~~~~ leapp.db entry for the PersistentNetNamesFacts is pasted below, in which the interfaces em4_X are the VF interfaces, which are present in the DB collected with RHEL 7.8, but this interface will not be available unitl the sriov_config service runs. ~~~~~~~~~~~ { "interfaces": [ { "devpath": "/devices/pci0000:16/0000:16:02.0/0000:18:00.0/net/em1", "driver": "i40e", "mac": "e4:43:4b:5c:96:80", "name": "em1", "pci_info": { "bus": "18", "device": "00", "domain": "0000", "function": "0" }, "vendor": "0x8086" }, { "devpath": "/devices/pci0000:16/0000:16:02.0/0000:18:00.1/net/em2", "driver": "i40e", "mac": "e4:43:4b:5c:96:81", "name": "em2", "pci_info": { "bus": "18", "device": "00", "domain": "0000", "function": "1" }, "vendor": "0x8086" }, { "devpath": "/devices/pci0000:16/0000:16:02.0/0000:18:00.3/net/em4", "driver": "i40e", "mac": "e4:43:4b:5c:96:83", "name": "em4", "pci_info": { "bus": "18", "device": "00", "domain": "0000", "function": "3" }, "vendor": "0x8086" }, { "devpath": "/devices/pci0000:16/0000:16:02.0/0000:18:0e.0/net/em4_0", "driver": "iavf", "mac": "1a:63:98:77:7b:81", "name": "em4_0", "pci_info": { "bus": "18", "device": "0e", "domain": "0000", "function": "0" }, "vendor": "0x8086" }, { "devpath": "/devices/pci0000:16/0000:16:02.0/0000:18:0e.1/net/em4_1", "driver": "iavf", "mac": "96:d5:f8:0f:81:6c", "name": "em4_1", "pci_info": { "bus": "18", "device": "0e", "domain": "0000", "function": "1" }, "vendor": "0x8086" }, { "devpath": "/devices/pci0000:16/0000:16:02.0/0000:18:0e.2/net/em4_2", "driver": "iavf", "mac": "5e:28:63:ba:7c:fe", "name": "em4_2", "pci_info": { "bus": "18", "device": "0e", "domain": "0000", "function": "2" }, "vendor": "0x8086" }, { "devpath": "/devices/pci0000:16/0000:16:02.0/0000:18:0e.3/net/em4_3", "driver": "iavf", "mac": "7e:35:bf:68:52:68", "name": "em4_3", "pci_info": { "bus": "18", "device": "0e", "domain": "0000", "function": "3" }, "vendor": "0x8086" }, { "devpath": "/devices/pci0000:64/0000:64:00.0/0000:65:00.0/net/p1p1", "driver": "mlx5_core", "mac": "98:03:9b:7f:9e:48", "name": "p1p1", "pci_info": { "bus": "65", "device": "00", "domain": "0000", "function": "0" }, "vendor": "0x15b3" }, { "devpath": "/devices/pci0000:64/0000:64:00.0/0000:65:00.1/net/p1p2", "driver": "mlx5_core", "mac": "98:03:9b:7f:9e:49", "name": "p1p2", "pci_info": { "bus": "65", "device": "00", "domain": "0000", "function": "1" }, "vendor": "0x15b3" } ] } ~~~~~~~~~~~ Ideally leapp should ignore the VF interfaces as these has to be created on every boot. It has to be discussed with leapp to for further inputs. A simple workaround would be to reset all the PFs of the compute node before the upgrade. For the leagacy SR-IOV, it will be simple to reset all the PFs to 0. But it will have additional complication with offload and nic paritioning, still analyzing for those features.
Validated the patch (https://review.opendev.org/#/c/745280) with FFU upgrade node and it is working. Before initiating the "leapp upgrade", during the "system_upgrade_prepare" step, all the VFs will be reset by setting '0' to sriov_numvfs for all the interfaces on the compute node. IMPACT - Before doing an upgrade run for a specific compute nodes, VMs should be brought down as there will be VFs attached to VM, by resetting it, the device is removed and it will create unexpected behavior on the VM. When the leapp upgrade prepares the DB, it will aggregate only the PFs, as there are no VFs. And on reboot, leapp will find all the interfaces and the upgrade will be successful. It avoids the hang issue. ~~~~ This patch also removes the "sriov_config.service" (installed by os-net-config), because after the ugprade to RHEL 8.2 the executable of the service "/usr/bin/os-net-config-sriov" will not be found, because it will be installed with os-net-config. Keep this service will result in an unwanted service failure during the boot process. The purpose of the service is to create the VFs on every reboot. As the OSP is not installed yet, it will not be able to configure. As part of the upgrade run at step 3, os-net-config will be run with --no-activate option, which will trigger the VFs creation and also restore the sriov_config.service, which will be used for subsequent reboots. ~~~~~~ This will create a problem for nic partitioning feature and ovs hardware offload. For nic paritionioning, the VFs will be used for an openstack network (InternalApi, Storage, Tenant). When the ifcfg scripts for the respective network is running, the underlying VF interface, will not be available. This will result in network.service failure. Also in order to restore the networks with VFs, os-net-config has to be run without "--no-activate". The same applies for offload feature as well, ifcfg run will expect the interface has been set to switchdev mode already, since sriov_config service didn't run, it will not be applied, These scenarios has to be analyzed further, I will create another BZ to validate those features with FFU.
RHOS-16.1-RHEL-8-20200821.n.0 Deployed 13 latest_cdn FFWD to 16.1, computes are up and accessible
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenStack Platform 16.1 director bug fix advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:3542