Bug 2168962
| Summary: | [17.1] IP addresses not configured in interfaces after running os-net-config | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Miguel Angel Nieto <mnietoji> |
| Component: | os-net-config | Assignee: | Vijayalakshmi Candappa <vcandapp> |
| Status: | CLOSED NOTABUG | QA Contact: | Eran Kuris <ekuris> |
| Severity: | high | Docs Contact: | |
| Priority: | medium | ||
| Version: | 17.1 (Wallaby) | CC: | bfournie, ekuris, hakhande, hbrock, jraju, jslagle, mburns, rjarry, supadhya, vcandapp |
| Target Milestone: | --- | Keywords: | Triaged |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-02-27 11:30:51 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
@mnietoji , I am trying to reproduce this issue in our dev lab - was this issue noticed in hw-offload setup or other regular depoyments like DPDK/SRIOV ? @vcandapp, I think it is related with hwoffload, I have not seen anyone reporting something similar. I will upload templates @mnietoji , are these nic mapping correct:
eg. for hostname: computehwoffload-r730
network_config:
template: /home/stack/ospd-17.1-geneve-ovn-dvr-hw-offload-ctlplane-dataplane-bonding-hybrid/nic-configs/computesriov.yaml
net_config_data_lookup:
nodegroup-dell-poweredge-r730:
dmiString: "system-product-name"
id: "PowerEdge R730"
nic1: "eno1" # In biosdevname "em1"
nic2: "eno2" # In biosdevname "em2"
nic3: "enp130s0f0" # In biosdevname "p4p1"
nic4: "enp130s0f1" # In biosdevname "p4p2"
nic5: "enp130s0f2" # In biosdevname "p4p3"
nic6: "enp130s0f3" # In biosdevname "p4p4"
nic7: "enp6s0f0" # In biosdevname "p7p1"
nic8: "enp6s0f1" # In biosdevname "p7p2"
nic9: "enp6s0f2" # In biosdevname "p7p3"
nic10: "enp6s0f3" # In biosdevname "p7p4"
nic11: "enp4s0f0np0" # In biosdevname "p6p1"
nic12: "enp4s0f1np1" # In biosdevname "p6p2"
[tripleo-admin@computehwoffload-r730 ~]$ ip a show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp6s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether f8:f2:1e:03:c8:c0 brd ff:ff:ff:ff:ff:ff
inet6 fe80::937d:c37a:b012:ffd6/64 scope link noprefixroute
valid_lft forever preferred_lft forever
3: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 80:18:44:f2:88:b0 brd ff:ff:ff:ff:ff:ff
altname enp1s0f0
inet6 2620:52:0:2eae:e398:bb57:4259:748a/64 scope global dynamic noprefixroute
valid_lft 2591932sec preferred_lft 604732sec
inet6 fe80::6681:3961:29f8:fdb3/64 scope link noprefixroute
valid_lft forever preferred_lft forever
4: enp6s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether f8:f2:1e:03:c8:c2 brd ff:ff:ff:ff:ff:ff
inet6 fe80::897c:83a8:57d2:1023/64 scope link noprefixroute
valid_lft forever preferred_lft forever
5: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 80:18:44:f2:88:b1 brd ff:ff:ff:ff:ff:ff
altname enp1s0f1
inet 192.0.80.8/24 brd 192.0.80.255 scope global dynamic noprefixroute eno2
valid_lft 86329sec preferred_lft 86329sec
inet6 fe80::8218:44ff:fef2:88b1/64 scope link
valid_lft forever preferred_lft forever
6: eno3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether 80:18:44:f2:88:b2 brd ff:ff:ff:ff:ff:ff
altname enp2s0f0
7: enp6s0f2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether f8:f2:1e:03:c8:c4 brd ff:ff:ff:ff:ff:ff
inet6 fe80::801:3bb3:8c9f:6b7b/64 scope link noprefixroute
valid_lft forever preferred_lft forever
8: eno4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether 80:18:44:f2:88:b3 brd ff:ff:ff:ff:ff:ff
altname enp2s0f1
9: enp6s0f3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether f8:f2:1e:03:c8:c6 brd ff:ff:ff:ff:ff:ff
inet6 fe80::f9a:16bf:cc53:1890/64 scope link noprefixroute
valid_lft forever preferred_lft forever
10: enp130s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether f8:f2:1e:03:61:a0 brd ff:ff:ff:ff:ff:ff
inet6 fe80::9d89:fea5:eba3:bb25/64 scope link noprefixroute
valid_lft forever preferred_lft forever
11: enp130s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether f8:f2:1e:03:61:a2 brd ff:ff:ff:ff:ff:ff
inet6 fe80::432a:e34e:4ad7:3b90/64 scope link noprefixroute
valid_lft forever preferred_lft forever
12: enp4s0f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether ec:0d:9a:7d:7d:2a brd ff:ff:ff:ff:ff:ff
inet6 fe80::17b6:c0e1:c42e:274d/64 scope link noprefixroute
valid_lft forever preferred_lft forever
13: enp130s0f2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether f8:f2:1e:03:61:a4 brd ff:ff:ff:ff:ff:ff
inet6 fe80::e19:f0c2:249d:2e7a/64 scope link noprefixroute
valid_lft forever preferred_lft forever
14: enp4s0f1np1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether ec:0d:9a:7d:7d:2b brd ff:ff:ff:ff:ff:ff
inet6 fe80::e205:6bf4:d4ed:5aac/64 scope link noprefixroute
valid_lft forever preferred_lft forever
15: enp8s0f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether b8:3f:d2:6f:ea:60 brd ff:ff:ff:ff:ff:ff
inet6 fe80::6c7a:56db:7f14:7772/64 scope link noprefixroute
valid_lft forever preferred_lft forever
16: enp130s0f3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether f8:f2:1e:03:61:a6 brd ff:ff:ff:ff:ff:ff
inet6 fe80::3b3a:5c02:20c7:be4/64 scope link noprefixroute
valid_lft forever preferred_lft forever
17: enp8s0f1np1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether b8:3f:d2:6f:ea:61 brd ff:ff:ff:ff:ff:ff
inet6 fe80::4319:cc82:c40:32bc/64 scope link noprefixroute
valid_lft forever preferred_lft forever
[tripleo-admin@computehwoffload-r730 ~]$
Miguel, also the BLS flag playbook (bls_workaround.yaml) is not required now (in the baremetal deployment file):
count: 1
instances:
- hostname: computehwoffload-r740
name: computeb
defaults:
networks:
- network: external
subnet: external_subnet
- network: internal_api
subnet: internal_api_subnet
- network: tenant
subnet: tenant_subnet
- network: storage
subnet: storage_subnet
network_config:
template: /home/stack/ospd-17.1-geneve-ovn-dvr-hw-offload-ctlplane-dataplane-bonding-hybrid/nic-configs/computesriov.yaml
net_config_data_lookup:
nodegroup-dell-poweredge-r740:
dmiString: "system-product-name"
id: "PowerEdge R740"
nic1: "eno1" # In biosdevname "em1"
nic2: "eno2" # In biosdevname "em2"
nic3: "ens4f0" # In biosdevname "p4p1"
nic4: "ens4f1" # In biosdevname "p4p2"
nic5: "ens4f2" # In biosdevname "p4p3"
nic6: "ens4f3" # In biosdevname "p4p4"
nic7: "ens7f0" # In biosdevname "p7p1"
nic8: "ens7f1" # In biosdevname "p7p2"
nic9: "ens7f2" # In biosdevname "p6p2"
nic10: "ens7f3" # In biosdevname "p6p2"
nic11: "ens6f0np0" # In biosdevname "p6p1"
nic12: "ens6f1np1" # In biosdevname "p6p1"
config_drive:
cloud_config:
ssh_pwauth: true
disable_root: false
chpasswd:
list: |-
root:12345678
expire: false
ansible_playbooks:
- playbook: /home/stack/ospd-17.1-geneve-ovn-dvr-hw-offload-ctlplane-dataplane-bonding-hybrid/playbooks/bz2158628.yaml
- playbook: /home/stack/ospd-17.1-geneve-ovn-dvr-hw-offload-ctlplane-dataplane-bonding-hybrid/playbooks/bls_workaround.yaml
- playbook: /usr/share/ansible/tripleo-playbooks/cli-overcloud-node-kernelargs.yaml
extra_vars:
kernel_args: "default_hugepagesz=1GB hugepagesz=1G hugepages=64 iommu=pt intel_iommu=on tsx=off isolcpus=2-19,22-39"
reboot_wait_timeout: 900
tuned_profile: "cpu-partitioning"
tuned_isolated_cores: "2-19,22-39"
After checking the templates, the external gateway IP was not rendered (empty) - which caused the safe default nic config to be applied by os-net-config. This triggered DHCP requests on all 12+ interfaces at the same time - leading to state Updating the nic templates will fix this issue - but will keep this BZ open to fix the render safe defaults issue Hi I fixed the gateway issue, there was a line missing in the templates with the gateway. But I think this is not causing the problem described with connectivity. This is the behaviour I see: 1. I provision servers 2. I connect to the computes and ping from one compute to the other one using the vlan that goes through the mellanox bond and it fails 3. If i try to reboot network service it does not solve the issue 4. if I reboot the computes, then ping starts working 5. If I remove the bonding and create regular interfaces, ping does not work either So, somehow, I think issue is related with mellanox nics, they needs a reboot to start working Hi, i have just realized that what I have described in my previous comment [1] is a different issue than the one described in the description of this bz So, fixing the templates, the original issue is solved. For the other issue described in [1], I think i will create a different bz For this bz, can I just close as NOTABUG? The problem was that the gateway line was missing in templates [1] https://bugzilla.redhat.com/show_bug.cgi?id=2168962#c8 (In reply to Miguel Angel Nieto from comment #9) > Hi, i have just realized that what I have described in my previous comment > [1] is a different issue than the one described in the description of this bz > > So, fixing the templates, the original issue is solved. > > For the other issue described in [1], I think i will create a different bz > > For this bz, can I just close as NOTABUG? The problem was that the gateway > line was missing in templates > > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=2168962#c8 Yes please close this one if this issue is resolved. Raise new one for the new behavior you observed. Thanks Due to the mis-configuration in the nic config, the os-net-config schema validation failed and triggered "safe_default" NIC config. Closing this BZ as "Not a Bug", but will create an enhancement BZ in launchpad to avoid trigger of dhcp_discover for larger # of interfaces during node provision. |
Description of problem: During deployment, provisioning stage fails with the following error: Ansible execution failed. playbook: /usr/share/ansible/tripleo-playbooks/cli-overcloud-node-network-config.yaml, Run Status: failed, Return Code: 2 Exception occured while running the command Traceback (most recent call last): File "/usr/lib/python3.9/site-packages/tripleoclient/command.py", line 32, in run super(Command, self).run(parsed_args) File "/usr/lib/python3.9/site-packages/osc_lib/command/command.py", line 39, in run return super(Command, self).run(parsed_args) File "/usr/lib/python3.9/site-packages/cliff/command.py", line 186, in run return_code = self.take_action(parsed_args) or 0 File "/usr/lib/python3.9/site-packages/tripleoclient/v2/overcloud_node.py", line 350, in take_action oooutils.run_role_playbooks(self, working_dir, roles_file_dir, File "/usr/lib/python3.9/site-packages/tripleoclient/utils.py", line 2893, in run_role_playbooks run_role_playbook(self, inventory, constants.ANSIBLE_TRIPLEO_PLAYBOOKS, File "/usr/lib/python3.9/site-packages/tripleoclient/utils.py", line 2848, in run_role_playbook run_ansible_playbook( File "/usr/lib/python3.9/site-packages/tripleoclient/utils.py", line 741, in run_ansible_playbook raise RuntimeError(err_msg) RuntimeError: Ansible execution failed. playbook: /usr/share/ansible/tripleo-playbooks/cli-overcloud-node-network-config.yaml, Run Status: failed, Return Code: 2 Ansible execution failed. playbook: /usr/share/ansible/tripleo-playbooks/cli-overcloud-node-network-config.yaml, Run Status: failed, Return Code: 2 ansible.log 2023-02-10 15:07:48,556 p=407073 u=stack n=ansible | 2023-02-10 15:07:48.555094 | 5254006a-b4cd-779a-f378-0000000001e2 | CHANGED | Render network_config from template | computehwoffload-r730 2023-02-10 15:07:48,651 p=407073 u=stack n=ansible | 2023-02-10 15:07:48.650367 | 5254006a-b4cd-779a-f378-0000000001e2 | CHANGED | Render network_config from template | controller-1 2023-02-10 15:07:48,653 p=407073 u=stack n=ansible | 2023-02-10 15:07:48.652812 | 5254006a-b4cd-779a-f378-0000000001e2 | CHANGED | Render network_config from template | controller-0 2023-02-10 15:07:48,686 p=407073 u=stack n=ansible | 2023-02-10 15:07:48.686433 | 5254006a-b4cd-779a-f378-0000000001e2 | CHANGED | Render network_config from template | controller-2 2023-02-10 15:07:48,872 p=407073 u=stack n=ansible | 2023-02-10 15:07:48.871920 | 5254006a-b4cd-779a-f378-0000000001e2 | CHANGED | Render network_config from template | computehwoffload-r740 2023-02-10 15:07:48,891 p=407073 u=stack n=ansible | 2023-02-10 15:07:48.890875 | 5254006a-b4cd-779a-f378-0000000001e3 | TASK | Run tripleo_os_net_config_module with network_config 2023-02-10 15:12:56,193 p=407073 u=stack n=ansible | 2023-02-10 15:12:56.192545 | 5254006a-b4cd-779a-f378-0000000001e3 | FATAL | Run tripleo_os_net_config_module with network_config | controller-1 | error={"ansible_job_id": "208820406518.2663", "changed": false, "child_pid": 2667, "finished": 1, "msg": "Timeout exceeded", "results_file": "/root/.ansible_async/208820406518.2663", "started": 1, "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2023-02-10 15:12:56,484 p=407073 u=stack n=ansible | 2023-02-10 15:12:56.484313 | 5254006a-b4cd-779a-f378-0000000001e3 | FATAL | Run tripleo_os_net_config_module with network_config | controller-0 | error={"ansible_job_id": "703380256130.2667", "changed": false, "child_pid": 2671, "finished": 1, "msg": "Timeout exceeded", "results_file": "/root/.ansible_async/703380256130.2667", "started": 1, "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2023-02-10 15:12:57,921 p=407073 u=stack n=ansible | 2023-02-10 15:12:57.920818 | 5254006a-b4cd-779a-f378-0000000001e3 | FATAL | Run tripleo_os_net_config_module with network_config | controller-2 | error={"ansible_job_id": "471278742925.2657", "changed": false, "child_pid": 2661, "finished": 1, "msg": "Timeout exceeded", "results_file": "/root/.ansible_async/471278742925.2657", "started": 1, "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2023-02-10 15:45:48,261 p=407073 u=stack n=ansible | 2023-02-10 15:45:48.260866 | 5254006a-b4cd-779a-f378-0000000001e3 | FATAL | Run tripleo_os_net_config_module with network_config | computehwoffload-r730 | error={"msg": "Failed to connect to the host via ssh: ssh: connect to host 192.0.80.12 port 22: No route to host"} 2023-02-10 15:49:09,280 p=407073 u=stack n=ansible | 2023-02-10 15:49:09.279761 | 5254006a-b4cd-779a-f378-0000000001e3 | FATAL | Run tripleo_os_net_config_module with network_config | computehwoffload-r740 | error={"msg": "Failed to connect to the host via ssh: ssh: connect to host 192.0.80.14 port 22: No route to host"} (undercloud) [stack@undercloud-0 ~]$ metalsmith list +--------------------------------------+--------------+--------------------------------------+-----------------------+--------+----------------------+ | UUID | Node Name | Allocation UUID | Hostname | State | IP Addresses | +--------------------------------------+--------------+--------------------------------------+-----------------------+--------+----------------------+ | c9b3cfee-4c0d-48b0-903a-3acf525c34ed | computea | 07e7ccda-17df-48f6-8470-c33550185d97 | computehwoffload-r730 | ACTIVE | ctlplane=192.0.80.12 | | 527004ea-7b47-4aef-bd51-56c2e8e4322a | computeb | fb8473a3-f009-4070-905b-da7157f86e4a | computehwoffload-r740 | ACTIVE | ctlplane=192.0.80.14 | | 4e860f1d-0746-4f1e-9100-66529b5c0c32 | controller-0 | 7dc87365-7e1d-424d-9229-5f3c7f589333 | controller-0 | ACTIVE | ctlplane=192.0.80.15 | | 56250202-1998-4ca9-aa6a-116972560645 | controller-1 | b2bd2834-a533-4224-9557-d3005cd35391 | controller-1 | ACTIVE | ctlplane=192.0.80.19 | | 36ee9931-48f6-4d5f-b793-19ae4502117d | controller-2 | 1cbdf346-d465-488b-b638-3cf7a419342e | controller-2 | ACTIVE | ctlplane=192.0.80.10 | +--------------------------------------+--------------+--------------------------------------+-----------------------+--------+----------------------+ There is no ping to computes --- 192.0.80.10 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 1.247/1.247/1.247/0.000 ms (undercloud) [stack@undercloud-0 ~]$ ping -c1 192.0.80.19 PING 192.0.80.19 (192.0.80.19) 56(84) bytes of data. 64 bytes from 192.0.80.19: icmp_seq=1 ttl=64 time=0.630 ms --- 192.0.80.19 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.630/0.630/0.630/0.000 ms (undercloud) [stack@undercloud-0 ~]$ ping -c1 192.0.80.15 PING 192.0.80.15 (192.0.80.15) 56(84) bytes of data. 64 bytes from 192.0.80.15: icmp_seq=1 ttl=64 time=0.982 ms --- 192.0.80.15 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.982/0.982/0.982/0.000 ms (undercloud) [stack@undercloud-0 ~]$ ping -c1 192.0.80.14 PING 192.0.80.14 (192.0.80.14) 56(84) bytes of data. From 192.0.80.1 icmp_seq=1 Destination Host Unreachable --- 192.0.80.14 ping statistics --- 1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms (undercloud) [stack@undercloud-0 ~]$ ping -c1 192.0.80.12 PING 192.0.80.12 (192.0.80.12) 56(84) bytes of data. --- 192.0.80.12 ping statistics --- 1 packets transmitted, 0 received, 100% packet loss, time 0ms If i check interfaces in computes, i can see that ip is not configured in interfaces. IPs are configured if I execute dhclient interface or if I reboot the compute. So, I think that in the os-net-config script must be missing one step to apply configuration. I attach sos report Version-Release number of selected component (if applicable): RHOS-17.1-RHEL-9-20230131.n.2 Red Hat Enterprise Linux release 9.2 Beta (Plow) os-net-config-14.2.1-1.20230109230810.d682470.el9ost.noarch How reproducible: Deploy osp17.1 templates Actual results: Provisioning stage fails Expected results: Provisioning stage should pass Additional info: