Description of problem: After running Overcloud Compute System Upgrade FFU OSP17 got this error on RHEL 9 compute: ~~~ [root@computedpdksriov-0 ~]# tail /var/log/containers/stdouts/nova_wait_for_compute_service.log.1 2023-07-27T23:48:32.489679495+00:00 stdout F return self._plugin.get_auth_ref(session, **kwargs) 2023-07-27T23:48:32.489679495+00:00 stdout F File "/usr/lib/python3.9/site-packages/keystoneauth1/identity/v3/base.py", line 187, in get_auth_ref 2023-07-27T23:48:32.489679495+00:00 stdout F resp = session.post(token_url, json=body, headers=headers, 2023-07-27T23:48:32.489679495+00:00 stdout F File "/usr/lib/python3.9/site-packages/keystoneauth1/session.py", line 1149, in post 2023-07-27T23:48:32.489679495+00:00 stdout F return self.request(url, 'POST', **kwargs) 2023-07-27T23:48:32.489679495+00:00 stdout F File "/usr/lib/python3.9/site-packages/keystoneauth1/session.py", line 931, in request 2023-07-27T23:48:32.489679495+00:00 stdout F resp = send(**kwargs) 2023-07-27T23:48:32.489679495+00:00 stdout F File "/usr/lib/python3.9/site-packages/keystoneauth1/session.py", line 1038, in _send_request 2023-07-27T23:48:32.489679495+00:00 stdout F raise exceptions.ConnectFailure(msg) 2023-07-27T23:48:32.489679495+00:00 stdout F keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://10.10.116.131:5000/v3/auth/tokens: HTTPConnectionPool(host='10.10.116.131', port=5000): Max retries exceeded with url: /v3/auth/tokens (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fabb53d6940>: Failed to establish a new connection: [Errno 113] No route to host')) ~~~ That error is caused because os-net-config ran with errors: ~~~ root@computedpdksriov-0 ~]# grep -i error /var/log/os-net-config.log 2023-07-27 23:21:43.047 ERROR os_net_config.objects.mapped_nics nic enp130s0f0 not found in available nics (eno1, eno2, eno3, eno4, enp5s0f0, enp5s0f1, enp5s0f2, enp5s0f3, enp130s0f2, enp130s0f3, p4p1, p4p2) 2023-07-27 23:21:43.048 ERROR os_net_config.objects.mapped_nics nic enp130s0f1 not found in available nics (eno1, eno2, eno3, eno4, enp5s0f0, enp5s0f1, enp5s0f2, enp5s0f3, enp130s0f2, enp130s0f3, p4p1, p4p2) 2023-07-27 23:21:43.777 ERROR os_net_config.common.interface_mac Unable to read mac address: nic3 2023-07-27 23:29:16.838 ERROR os_net_config.objects.mapped_nics nic enp130s0f0 not found in available nics (eno1, eno2, eno3, eno4, enp5s0f0, enp5s0f1, enp5s0f2, enp5s0f3, enp130s0f2, enp130s0f3, p4p1, p4p2) 2023-07-27 23:29:16.839 ERROR os_net_config.objects.mapped_nics nic enp130s0f1 not found in available nics (eno1, eno2, eno3, eno4, enp5s0f0, enp5s0f1, enp5s0f2, enp5s0f3, enp130s0f2, enp130s0f3, p4p1, p4p2) 2023-07-27 23:30:25.427 ERROR os_net_config.objects.mapped_nics nic enp130s0f0 not found in available nics (eno1, eno2, eno3, eno4, enp5s0f0, enp5s0f1, enp5s0f2, enp5s0f3, enp130s0f2, enp130s0f3, p4p1, p4p2) 2023-07-27 23:30:25.427 ERROR os_net_config.objects.mapped_nics nic enp130s0f1 not found in available nics (eno1, eno2, eno3, eno4, enp5s0f0, enp5s0f1, enp5s0f2, enp5s0f3, enp130s0f2, enp130s0f3, p4p1, p4p2) 2023-07-27 23:31:01.326 ERROR os_net_config.objects.mapped_nics nic enp130s0f0 not found in available nics (eno1, eno2, eno3, eno4, enp5s0f0, enp5s0f1, enp5s0f2, enp5s0f3, enp130s0f2, enp130s0f3, p4p1, p4p2) 2023-07-27 23:31:01.326 ERROR os_net_config.objects.mapped_nics nic enp130s0f1 not found in available nics (eno1, eno2, eno3, eno4, enp5s0f0, enp5s0f1, enp5s0f2, enp5s0f3, enp130s0f2, enp130s0f3, p4p1, p4p2) 2023-07-27 23:34:12.449 ERROR os_net_config.objects.mapped_nics nic enp130s0f0 not found in available nics (eno1, eno2, eno3, eno4, enp5s0f0, enp5s0f1, enp5s0f2, enp5s0f3, enp130s0f2, enp130s0f3, p4p1, p4p2) 2023-07-27 23:34:12.449 ERROR os_net_config.objects.mapped_nics nic enp130s0f1 not found in available nics (eno1, eno2, eno3, eno4, enp5s0f0, enp5s0f1, enp5s0f2, enp5s0f3, enp130s0f2, enp130s0f3, p4p1, p4p2) 2023-07-27 23:35:08.095 ERROR os_net_config.objects.mapped_nics nic enp130s0f0 not found in available nics (eno1, eno2, eno3, eno4, enp5s0f0, enp5s0f1, enp5s0f2, enp5s0f3, enp130s0f2, enp130s0f3, p4p1, p4p2) 2023-07-27 23:35:08.095 ERROR os_net_config.objects.mapped_nics nic enp130s0f1 not found in available nics (eno1, eno2, eno3, eno4, enp5s0f0, enp5s0f1, enp5s0f2, enp5s0f3, enp130s0f2, enp130s0f3, p4p1, p4p2) ~~~ When running os-config-net manually we can see this: ~~~ [root@computedpdksriov-0 ~]# os-net-config --noop -c /etc/os-net-config/config.json 2023-07-28 08:10:05.670 ERROR os_net_config.objects.mapped_nics nic enp130s0f0 not found in available nics (eno1, eno2, eno3, eno4, enp5s0f0, enp5s0f1, enp5s0f2, enp5s0f3, enp130s0f2, enp130s0f3, p4p1, p4p2) 2023-07-28 08:10:05.670 ERROR os_net_config.objects.mapped_nics nic enp130s0f1 not found in available nics (eno1, eno2, eno3, eno4, enp5s0f0, enp5s0f1, enp5s0f2, enp5s0f3, enp130s0f2, enp130s0f3, p4p1, p4p2) 2023-07-28 08:10:05.683 WARNING os_net_config.objects.mapped_nics no mapping for interface p4p2 because nic10 is mapped to enp5s0f3 2023-07-28 08:10:05.683 WARNING os_net_config.objects.mapped_nics no mapping for interface p4p1 because nic9 is mapped to enp5s0f2 2023-07-28 08:10:05.687 ERROR os_net_config.common.interface_mac Unable to read mac address: nic3 Traceback (most recent call last): File "/bin/os-net-config", line 10, in <module> sys.exit(main()) File "/usr/lib/python3.9/site-packages/os_net_config/cli.py", line 335, in main provider.add_object(obj) File "/usr/lib/python3.9/site-packages/os_net_config/__init__.py", line 88, in add_object self.add_linux_bond(obj) File "/usr/lib/python3.9/site-packages/os_net_config/impl_ifcfg.py", line 890, in add_linux_bond data = self._add_common(bond) File "/usr/lib/python3.9/site-packages/os_net_config/impl_ifcfg.py", line 529, in _add_common primary_mac = common.interface_mac(primary_name) File "/usr/lib/python3.9/site-packages/os_net_config/common.py", line 180, in interface_mac with open(get_dev_path(name, '_address'), 'r') as f: FileNotFoundError: [Errno 2] No such file or directory: '/sys/class/net/nic3/address' The reason for that error is that the biosdevname is used for some interfaces: ~~~ [root@computedpdksriov-0 ~]# ls /sys/class/net/ bond_api/ br-dpdk1/ eno1/ eno4/ lo/ p4p2/ p7p3_2/ p7p3_5/ p7p3_8/ p7p4_1/ p7p4_4/ p7p4_7/ vlan516/ bonding_masters br-int/ eno2/ enp5s0f2/ ovs-netdev/ p7p3_0/ p7p3_3/ p7p3_6/ p7p3_9/ p7p4_2/ p7p4_5/ p7p4_8/ vlan518/ br-dpdk0/ br-link0/ eno3/ enp5s0f3/ p4p1/ p7p3_1/ p7p3_4/ p7p3_7/ p7p4_0/ p7p4_3/ p7p4_6/ p7p4_9/ ~~~ And we found that biosdevname=1 set in /proc/cmdline: ~~~ [root@computedpdksriov-0 ~]# cat /proc/cmdline BOOT_IMAGE=(hd0,gpt3)/boot/vmlinuz-5.14.0-284.24.1.el9_2.x86_64 root=UUID=2248aa8e-f026-4492-b8fe-79b8cdcab92b ro console=ttyS0 console=ttyS0,115200n81 no_timer_check rhgb quiet default_hugepagesz=1GB hugepagesz=1G hugepages=64 iommu=pt intel_iommu=on tsx=off isolcpus=2-19,22-39 skew_tick=1 nohz=on nohz_full=2-19,22-39 rcu_nocbs=2-19,22-39 tuned.non_isolcpus=00300003 intel_pstate=disable nosoftlockup rd.plymouth=0 plymouth.enable=0 crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M skew_tick=1 nohz=on nohz_full=2-19,22-39 rcu_nocbs=2-19,22-39 tuned.non_isolcpus=00300003 intel_pstate=disable nosoftlockup biosdevname=1 ~~~ After removing the biosdevname=1 option: ~~~ [root@computedpdksriov-0 ~]# cat /proc/cmdline BOOT_IMAGE=(hd0,gpt3)/boot/vmlinuz-5.14.0-284.24.1.el9_2.x86_64 root=UUID=2248aa8e-f026-4492-b8fe-79b8cdcab92b ro console=ttyS0 console=ttyS0,115200n81 no_timer_check rhgb quiet default_hugepagesz=1GB hugepagesz=1G hugepages=64 iommu=pt intel_iommu=on tsx=off isolcpus=2-19,22-39 skew_tick=1 nohz=on nohz_full=2-19,22-39 rcu_nocbs=2-19,22-39 tuned.non_isolcpus=00300003 intel_pstate=disable nosoftlockup rd.plymouth=0 plymouth.enable=0 crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M skew_tick=1 nohz=on nohz_full=2-19,22-39 rcu_nocbs=2-19,22-39 tuned.non_isolcpus=00300003 intel_pstate=disable nosoftlockup ~~~ The problem seems to be solved: no error when running os-net-config. And interfaces seems to have the Linux predictable names (required by os-net-config): ~~~[root@computedpdksriov-0 ~]# ls /sys/class/net/ bond_api br-dpdk0 br-int eno1 eno3 enp130s0f0 enp5s0f2 enp5s0f2v1 enp5s0f2v3 enp5s0f2v5 enp5s0f2v7 enp5s0f2v9 enp5s0f3v0 enp5s0f3v2 enp5s0f3v4 enp5s0f3v6 enp5s0f3v8 lo vlan516 bonding_masters br-dpdk1 br-link0 eno2 eno4 enp130s0f1 enp5s0f2v0 enp5s0f2v2 enp5s0f2v4 enp5s0f2v6 enp5s0f2v8 enp5s0f3 enp5s0f3v1 enp5s0f3v3 enp5s0f3v5 enp5s0f3v7 enp5s0f3v9 ovs-netdev vlan518 ~~~ Version-Release number of selected component (if applicable): 17.1 FFU (undercloud) [stack@undercloud-0 ~]$ cat core_puddle_version RHOS-17.1-RHEL-8-20230726.n.1 How reproducible: 100% Steps to Reproduce: 1. Run Overcloud Compute System Upgrade FFU OSP17 2. 3. Actual results: Expected results: Additional info: