Description of problem: Node provision is not failing when os-net-config nmstate provider rolls back to initial state (when triple_network_config is applied) Version-Release number of selected component (if applicable): 17.1.4 How reproducible: Always Steps to Reproduce: 1. openstack overcloud node provision --stack overcloud --network-config -y <baremetal_template> 2. 3. Actual results: Node provision command summary is as follows: PLAY RECAP ********************************************************************* computeovsdpdksriov-0 : ok=18 changed=7 unreachable=0 failed=0 skipped=7 rescued=0 ignored=0 controller-0 : ok=19 changed=8 unreachable=0 failed=0 skipped=6 rescued=0 ignored=0 controller-1 : ok=19 changed=8 unreachable=0 failed=0 skipped=6 rescued=0 ignored=0 controller-2 : ok=19 changed=8 unreachable=0 failed=0 skipped=6 rescued=0 ignored=0 2024-09-20 11:25:35.960495 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-09-20 11:25:35.960852 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Total Tasks: 26 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-09-20 11:25:35.961195 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Elapsed Time: 0:00:34.485244 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-09-20 11:25:35.961537 | UUID | Info | Host | Task Name | Run Time 2024-09-20 11:25:35.961857 | 525400d4-6bfb-7322-72ff-00000000001d | SUMMARY | controller-1 | Wait for provisioned nodes to boot | 10.77s 2024-09-20 11:25:35.962186 | 525400d4-6bfb-7322-72ff-00000000001d | SUMMARY | controller-0 | Wait for provisioned nodes to boot | 10.77s 2024-09-20 11:25:35.962523 | 525400d4-6bfb-7322-72ff-00000000001d | SUMMARY | computeovsdpdksriov-0 | Wait for provisioned nodes to boot | 10.76s 2024-09-20 11:25:35.962860 | 525400d4-6bfb-7322-72ff-00000000001d | SUMMARY | controller-2 | Wait for provisioned nodes to boot | 10.73s 2024-09-20 11:25:35.963198 | 525400d4-6bfb-7322-72ff-00000000019d | SUMMARY | controller-0 | tripleo_network_config : Run tripleo_os_net_config_module with network_config | 8.65s 2024-09-20 11:25:35.963539 | 525400d4-6bfb-7322-72ff-00000000019d | SUMMARY | computeovsdpdksriov-0 | tripleo_network_config : Run tripleo_os_net_config_module with network_config | 8.62s 2024-09-20 11:25:35.963854 | 525400d4-6bfb-7322-72ff-00000000019d | SUMMARY | controller-1 | tripleo_network_config : Run tripleo_os_net_config_module with network_config | 8.58s 2024-09-20 11:25:35.964203 | 525400d4-6bfb-7322-72ff-00000000019d | SUMMARY | controller-2 | tripleo_network_config : Run tripleo_os_net_config_module with network_config | 8.52s 2024-09-20 11:25:35.964538 | 525400d4-6bfb-7322-72ff-0000000000c2 | SUMMARY | controller-1 | tripleo_bootstrap : Ensure packages are actually well installed | 2.95s 2024-09-20 11:25:35.964862 | 525400d4-6bfb-7322-72ff-0000000000c2 | SUMMARY | controller-2 | tripleo_bootstrap : Ensure packages are actually well installed | 2.92s 2024-09-20 11:25:35.965193 | 525400d4-6bfb-7322-72ff-0000000000c2 | SUMMARY | controller-0 | tripleo_bootstrap : Ensure packages are actually well installed | 2.91s 2024-09-20 11:25:35.965543 | 525400d4-6bfb-7322-72ff-0000000000c2 | SUMMARY | computeovsdpdksriov-0 | tripleo_bootstrap : Ensure packages are actually well installed | 2.38s 2024-09-20 11:25:35.965866 | 525400d4-6bfb-7322-72ff-00000000000c | SUMMARY | computeovsdpdksriov-0 | tripleo_network_config : Create fcontext entry for tripleoconfig | 1.28s 2024-09-20 11:25:35.966216 | 525400d4-6bfb-7322-72ff-00000000019c | SUMMARY | controller-1 | tripleo_network_config : Render network_config from template | 1.26s 2024-09-20 11:25:35.966537 | 525400d4-6bfb-7322-72ff-00000000019c | SUMMARY | controller-0 | tripleo_network_config : Render network_config from template | 1.26s 2024-09-20 11:25:35.966871 | 525400d4-6bfb-7322-72ff-00000000019c | SUMMARY | computeovsdpdksriov-0 | tripleo_network_config : Render network_config from template | 1.26s 2024-09-20 11:25:35.967182 | 525400d4-6bfb-7322-72ff-00000000019c | SUMMARY | controller-2 | tripleo_network_config : Render network_config from template | 1.24s 2024-09-20 11:25:35.967511 | 525400d4-6bfb-7322-72ff-00000000001a | SUMMARY | controller-0 | tripleo_network_config : Disable cloud-init network config | 1.19s 2024-09-20 11:25:35.967804 | 525400d4-6bfb-7322-72ff-000000000019 | SUMMARY | controller-0 | tripleo_network_config : Write rc of NetworkConfig script | 1.19s 2024-09-20 11:25:35.968100 | 525400d4-6bfb-7322-72ff-000000000019 | SUMMARY | controller-1 | tripleo_network_config : Write rc of NetworkConfig script | 1.19s 2024-09-20 11:25:35.968465 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ End Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Nodes deployed successfully, add /home/stack/templates/overcloud-baremetal-deployed.yaml to your deployment environment (undercloud) [stack@undercloud-0 ~]$ But the network_config was not applied, as seen from os-net-logs 2024-09-20 07:25:31.808 ERROR os_net_config.impl_nmstate.apply Failure(s) occurred when applying configuration 2024-09-20 07:25:31.809 INFO os_net_config.impl_nmstate.rollback_to_initial_settings Rolling back to initial settings. 2024-09-20 07:25:31.868 INFO os_net_config.impl_nmstate.__dump_key_config ---------------------------- 2024-09-20 07:25:31.869 INFO os_net_config.impl_nmstate.__dump_key_config Applying the difference to go back to initial settings interfaces: Expected results: If Roll back happens during node provision, os-net-config should throw some sort of error/exception Additional info:
Now, when an error happens during deployment in os-net-config, and error is reported and the deployment stops. This is the new behaviour and it works as expected. In the past it only reported a warning and the deployment continued with the error. I introduced and error in os-net-config config file (remove numvfs in pf) and reprovision. I got and error reporting it openstack overcloud delete -b /home/stack/ospd-17.1-geneve-ovn-dpdk-sriov-ctlplane-dataplane-bonding-hybrid/network/baremetal_deployment.yaml --network-ports overcloud openstack overcloud node provision --stack overcloud --network-config -y /home/stack/ospd-17.1-geneve-ovn-dpdk-sriov-ctlplane-dataplane-bonding-hybrid/network/baremetal_deployment.yaml os-net-config-error 2024-10-21 11:03:21.020 ERROR os_net_config.main Config file failed schema validation at network_config/10: {'type': 'sriov_pf', 'name': 'nic11', 'mtu': 9000, 'use_dhcp': False, 'defroute': False, 'nm_controlled': True, 'hotplug': True, 'promisc': False, 'nic_mapping': {'nic1': 'eno1', 'nic10': 'enp6s0f3', 'nic11': 'enp4s0f0np0', 'nic12': 'enp4s0f1np1', 'nic2': 'eno2', 'nic3': 'enp130s0f0', 'nic4': 'enp130s0f1', 'nic5': 'enp130s0f2', 'nic6': 'enp130s0f3', 'nic7': 'enp6s0f0', 'nic8': 'enp6s0f1', 'nic9': 'enp6s0f2'}, 'persist_mapping': False} is not valid under any of the given schemas Sub-schemas tested and not matching: - items/oneOf/sriov_pf/required: 'numvfs' is a required property deployment error 2024-10-22 13:53:09.270828 | 525400a7-f449-eb76-8061-0000000001b5 | TASK | Run tripleo_os_net_config_module with network_config ASYNC POLL on controller-0: jid=750224632992.2652 started=1 finished=0 ASYNC POLL on controller-1: jid=121260632496.2644 started=1 finished=0 ASYNC POLL on controller-2: jid=88157384306.2638 started=1 finished=0 ASYNC FAILED on compute-r730: jid=894469167115.3400 2024-10-22 13:53:14.026933 | 525400a7-f449-eb76-8061-0000000001b5 | FATAL | Run tripleo_os_net_config_module with network_config | compute-r730 | error={"ansible_job_id": "894469167115.3400", "changed": false, "finished": 1, "msg": " Running os-net-config --config-file /etc/os-net-config/config.yaml --debug --detailed-exit-codes --provider nmstate failed with return code 1.", "rc": 1, "results_file": "/root/.ansible_async/894469167115.3400", "started": 1, "stderr": "" , "stderr_lines": [], "stdout": "2024-10-22 09:53:11.030 INFO os_net_config.main Using config file at: /etc/os-net-config/config.yaml\n2024-10-22 09:53:11.030 INFO os_net_config.main Using mapping file at: /etc/os-net-config/mapping.yaml\ n2024-10-22 09:53:11.032 DEBUG os_net_config.main interface_mapping: {'nic1': 'eno1', 'nic10': 'enp6s0f3', 'nic11': 'enp4s0f0np0', 'nic12': 'enp4s0f1np1', 'nic2': 'eno2', 'nic3': 'enp130s0f0', 'nic4': 'enp130s0f1', 'nic5': 'enp130s0f2', ' nic6': 'enp130s0f3', 'nic7': 'enp6s0f0', 'nic8': 'enp6s0f1', 'nic9': 'enp6s0f2'}\n2024-10-22 09:53:11.032 DEBUG os_net_config.main persist_mapping: False\n2024-10-22 09:53:11.050 DEBUG os_net_config.main network_config: [{'type': 'interfa ce', 'name': 'nic1', 'use_dhcp': False, 'defroute': False}, {'type': 'interface', 'name': 'nic2', 'use_dhcp': False, 'addresses': [{'ip_netmask': '192.0.80.22/24'}], 'routes': [{'default': True, 'next_hop': '192.0.80.1'}]}, {'type': 'linu x_bond', 'name': 'bond_api', 'use_dhcp': False, 'bonding_options': 'mode=active-backup', 'dns_servers': ['10.47.242.10', '10.45.248.15'], 'members': [{'type': 'interface', 'name': 'nic3', 'primary': True}, {'type': 'interface', 'name': 'n ic4'}]}, {'type': 'vlan', 'vlan_id': 140, 'device': 'bond_api', 'addresses': [{'ip_netmask': '10.10.140.145/24'}]}, {'type': 'vlan', 'vlan_id': 142, 'device': 'bond_api', 'addresses': [{'ip_netmask': '10.10.142.139/24'}]}, {'type': 'ovs_u ser_bridge', 'name': 'br-link0', 'use_dhcp': False, 'ovs_extra': 'set port br-link0 tag=141', 'addresses': [{'ip_netmask': '10.10.141.115/24'}], 'members': [{'type': 'ovs_dpdk_bond', 'name': 'dpdkbond0', 'ovs_options': 'bond_mode=balance- slb', 'mtu': 9000, 'rx_queue': 1, 'members': [{'type': 'ovs_dpdk_port', 'name': 'dpdk0', 'rx_queue_size': 2048, 'tx_queue_size': 2048, 'members': [{'type': 'interface', 'name': 'nic7'}]}, {'type': 'ovs_dpdk_port', 'name': 'dpdk1', 'rx_que ue_size': 2048, 'tx_queue_size': 2048, 'members': [{'type': 'interface', 'name': 'nic8'}]}]}]}, {'type': 'ovs_user_bridge', 'name': 'br-dpdk0', 'use_dhcp': False, 'members': [{'type': 'ovs_dpdk_port', 'mtu': 9000, 'rx_queue': 1, 'rx_queue _size': 2048, 'tx_queue_size': 2048, 'name': 'dpdk2', 'members': [{'type': 'interface', 'name': 'nic5'}]}]}, {'type': 'ovs_user_bridge', 'name': 'br-dpdk1', 'use_dhcp': False, 'members': [{'type': 'ovs_dpdk_port', 'mtu': 9000, 'rx_queue': 1, 'rx_queue_size': 2048, 'tx_queue_size': 2048, 'name': 'dpdk3', 'members': [{'type': 'interface', 'name': 'nic6'}]}]}, {'type': 'sriov_pf', 'name': 'nic9', 'mtu': 9000, 'numvfs': 4, 'use_dhcp': False, 'defroute': False, 'nm_controlled' : True, 'hotplug': True, 'promisc': False}, {'type': 'sriov_pf', 'name': 'nic10', 'mtu': 9000, 'numvfs': 4, 'use_dhcp': False, 'defroute': False, 'nm_controlled': True, 'hotplug': True, 'promisc': False}, {'type': 'sriov_pf', 'name': 'nic 11', 'mtu': 9000, 'use_dhcp': False, 'defroute': False, 'nm_controlled': True, 'hotplug': True, 'promisc': False}, {'type': 'sriov_pf', 'name': 'nic12', 'mtu': 9000, 'use_dhcp': False, 'defroute': False, 'nm_controlled': True, 'hotplug': True, 'promisc': False}]\n2024-10-22 09:53:11.370 ERROR os_net_config.main Config file failed schema validation at network_config/10:\n {'type': 'sriov_pf', 'name': 'nic11', 'mtu': 9000, 'use_dhcp': False, 'defroute': False, 'nm_contro lled': True, 'hotplug': True, 'promisc': False, 'nic_mapping': {'nic1': 'eno1', 'nic10': 'enp6s0f3', 'nic11': 'enp4s0f0np0', 'nic12': 'enp4s0f1np1', 'nic2': 'eno2', 'nic3': 'enp130s0f0', 'nic4': 'enp130s0f1', 'nic5': 'enp130s0f2', 'nic6': 'enp130s0f3', 'nic7': 'enp6s0f0', 'nic8': 'enp6s0f1', 'nic9': 'enp6s0f2'}, 'persist_mapping': False} is not valid under any of the given schemas\n Sub-schemas tested and not matching:\n - items/oneOf/sriov_pf/required: 'numvfs' is a re quired property\nConfig file failed schema validation at network_config/11:\n {'type': 'sriov_pf', 'name': 'nic12', 'mtu': 9000, 'use_dhcp': False, 'defroute': False, 'nm_controlled': True, 'hotplug': True, 'promisc': False, 'nic_mappi ng': {'nic1': 'eno1', 'nic10': 'enp6s0f3', 'nic11': 'enp4s0f0np0', 'nic12': 'enp4s0f1np1', 'nic2': 'eno2', 'nic3': 'enp130s0f0', 'nic4': 'enp130s0f1', 'nic5': 'enp130s0f2', 'nic6': 'enp130s0f3', 'nic7': 'enp6s0f0', 'nic8': 'enp6s0f1', 'ni c9': 'enp6s0f2'}, 'persist_mapping': False} is not valid under any of the given schemas\n Sub-schemas tested and not matching:\n - items/oneOf/sriov_pf/required: 'numvfs' is a required property\n", "stdout_lines": ["2024-10-22 09:53:11. 030 INFO os_net_config.main Using config file at: /etc/os-net-config/config.yaml", "2024-10-22 09:53:11.030 INFO os_net_config.main Using mapping file at: /etc/os-net-config/mapping.yaml", "2024-10-22 09:53:11.032 DEBUG os_net_config.main interface_mapping: {'nic1': 'eno1', 'nic10': 'enp6s0f3', 'nic11': 'enp4s0f0np0', 'nic12': 'enp4s0f1np1', 'nic2': 'eno2', 'nic3': 'enp130s0f0', 'nic4': 'enp130s0f1', 'nic5': 'enp130s0f2', 'nic6': 'enp130s0f3', 'nic7': 'enp6s0f0', 'nic8': 'enp6s0f1', 'nic9': 'enp6s0f2'}", "2024-10-22 09:53:11.032 DEBUG os_net_config.main persist_mapping: False", "2024-10-22 09:53:11.050 DEBUG os_net_config.main network_config: [{'type': 'interface', 'name': 'nic1', 'use_dhcp': False, 'defr oute': False}, {'type': 'interface', 'name': 'nic2', 'use_dhcp': False, 'addresses': [{'ip_netmask': '192.0.80.22/24'}], 'routes': [{'default': True, 'next_hop': '192.0.80.1'}]}, {'type': 'linux_bond', 'name': 'bond_api', 'use_dhcp': Fals e, 'bonding_options': 'mode=active-backup', 'dns_servers': ['10.47.242.10', '10.45.248.15'], 'members': [{'type': 'interface', 'name': 'nic3', 'primary': True}, {'type': 'interface', 'name': 'nic4'}]}, {'type': 'vlan', 'vlan_id': 140, 'de vice': 'bond_api', 'addresses': [{'ip_netmask': '10.10.140.145/24'}]}, {'type': 'vlan', 'vlan_id': 142, 'device': 'bond_api', 'addresses': [{'ip_netmask': '10.10.142.139/24'}]}, {'type': 'ovs_user_bridge', 'name': 'br-link0', 'use_dhcp': False, 'ovs_extra': 'set port br-link0 tag=141', 'addresses': [{'ip_netmask': '10.10.141.115/24'}], 'members': [{'type': 'ovs_dpdk_bond', 'name': 'dpdkbond0', 'ovs_options': 'bond_mode=balance-slb', 'mtu': 9000, 'rx_queue': 1, 'members': [{'type': 'ovs_dpdk_port', 'name': 'dpdk0', 'rx_queue_size': 2048, 'tx_queue_size': 2048, 'members': [{'type': 'interface', 'name': 'nic7'}]}, {'type': 'ovs_dpdk_port', 'name': 'dpdk1', 'rx_queue_size': 2048, 'tx_queue_size': 2048, 'membe rs': [{'type': 'interface', 'name': 'nic8'}]}]}]}, {'type': 'ovs_user_bridge', 'name': 'br-dpdk0', 'use_dhcp': False, 'members': [{'type': 'ovs_dpdk_port', 'mtu': 9000, 'rx_queue': 1, 'rx_queue_size': 2048, 'tx_queue_size': 2048, 'name': 'dpdk2', 'members': [{'type': 'interface', 'name': 'nic5'}]}]}, {'type': 'ovs_user_bridge', 'name': 'br-dpdk1', 'use_dhcp': False, 'members': [{'type': 'ovs_dpdk_port', 'mtu': 9000, 'rx_queue': 1, 'rx_queue_size': 2048, 'tx_queue_size': 2 048, 'name': 'dpdk3', 'members': [{'type': 'interface', 'name': 'nic6'}]}]}, {'type': 'sriov_pf', 'name': 'nic9', 'mtu': 9000, 'numvfs': 4, 'use_dhcp': False, 'defroute': False, 'nm_controlled': True, 'hotplug': True, 'promisc': False}, { 'type': 'sriov_pf', 'name': 'nic10', 'mtu': 9000, 'numvfs': 4, 'use_dhcp': False, 'defroute': False, 'nm_controlled': True, 'hotplug': True, 'promisc': False}, {'type': 'sriov_pf', 'name': 'nic11', 'mtu': 9000, 'use_dhcp': False, 'defrout e': False, 'nm_controlled': True, 'hotplug': True, 'promisc': False}, {'type': 'sriov_pf', 'name': 'nic12', 'mtu': 9000, 'use_dhcp': False, 'defroute': False, 'nm_controlled': True, 'hotplug': True, 'promisc': False}]", "2024-10-22 09:53: 11.370 ERROR os_net_config.main Config file failed schema validation at network_config/10:", " {'type': 'sriov_pf', 'name': 'nic11', 'mtu': 9000, 'use_dhcp': False, 'defroute': False, 'nm_controlled': True, 'hotplug': True, 'promisc': False, 'nic_mapping': {'nic1': 'eno1', 'nic10': 'enp6s0f3', 'nic11': 'enp4s0f0np0', 'nic12': 'enp4s0f1np1', 'nic2': 'eno2', 'nic3': 'enp130s0f0', 'nic4': 'enp130s0f1', 'nic5': 'enp130s0f2', 'nic6': 'enp130s0f3', 'nic7': 'enp6s0f0', 'nic8' : 'enp6s0f1', 'nic9': 'enp6s0f2'}, 'persist_mapping': False} is not valid under any of the given schemas", " Sub-schemas tested and not matching:", " - items/oneOf/sriov_pf/required: 'numvfs' is a required property", "Config file failed schema validation at network_config/11:", " {'type': 'sriov_pf', 'name': 'nic12', 'mtu': 9000, 'use_dhcp': False, 'defroute': False, 'nm_controlled': True, 'hotplug': True, 'promisc': False, 'nic_mapping': {'nic1': 'eno1', 'nic10': 'e np6s0f3', 'nic11': 'enp4s0f0np0', 'nic12': 'enp4s0f1np1', 'nic2': 'eno2', 'nic3': 'enp130s0f0', 'nic4': 'enp130s0f1', 'nic5': 'enp130s0f2', 'nic6': 'enp130s0f3', 'nic7': 'enp6s0f0', 'nic8': 'enp6s0f1', 'nic9': 'enp6s0f2'}, 'persist_mappin g': False} is not valid under any of the given schemas", " Sub-schemas tested and not matching:", " - items/oneOf/sriov_pf/required: 'numvfs' is a required property"]} 2024-10-22 13:53:14.028014 | 525400a7-f449-eb76-8061-0000000001b5 | TIMING | tripleo_network_config : Run tripleo_os_net_config_module with network_config | compute-r730 | 0:00:31.172492 | 4.75s ASYNC FAILED on compute-r740: jid=37400963538.3395 I had to set this variable to false in /usr/share/ansible/roles/tripleo_network_config/defaults/main.yml tripleo_network_config_safe_defaults: false Otherwise "tripleo_os_net_config_module with network_config" does not fail and the deployment fails later on because there is no connectivity to the compute node. @viji, should this variable set to false by default?
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (RHOSP 17.1.4 bug fix and enhancement advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2024:9974