Created attachment 1664832 [details] ansible.log Description of problem: OSP 16 DCN with IPv6 and Spine-Leaf network topology deployment failed on 2020-02-21 18:49:19,355 p=885 u=mistral | fatal: [central-controller0-0]: FAILED! => {"changed": true, "cmd": ["chronyc", "waitsync", "20"], "del ta": "0:03:10.191617", "end": "2020-02-21 13:49:19.283560", "msg": "non-zero return code", "rc": 1, "start": "2020-02-21 13:46:09.091943", "stderr ": "", "stderr_lines": [], "stdout": "try: 1, refid: 00000000, correction: 0.000000000, skew: 0.000\ntry: 2, refid: 00000000, correction: 0.000000 000, skew: 0.000\ntry: 3, refid: 00000000, correction: 0.000000000, skew: 0.000\ntry: 4, refid: 00000000, correction: 0.000000000, skew: 0.000\ntr y: 5, refid: 00000000, correction: 0.000000000, skew: 0.000\ntry: 6, refid: 00000000, correction: 0.000000000, skew: 0.000\ntry: 7, refid: 0000000 0, correction: 0.000000001, skew: 0.000\ntry: 8, refid: 00000000, correction: 0.000000001, skew: 0.000\ntry: 9, refid: 00000000, correction: 0.000 000001, skew: 0.000\ntry: 10, refid: 00000000, correction: 0.000000001, skew: 0.000\ntry: 11, refid: 00000000, correction: 0.000000001, skew: 0.00 0\ntry: 12, refid: 00000000, correction: 0.000000001, skew: 0.000\ntry: 13, refid: 00000000, correction: 0.000000001, skew: 0.000\ntry: 14, refid: 00000000, correction: 0.000000001, skew: 0.000\ntry: 15, refid: 00000000, correction: 0.000000001, skew: 0.000\ntry: 16, refid: 00000000, correct ion: 0.000000001, skew: 0.000\ntry: 17, refid: 00000000, correction: 0.000000002, skew: 0.000\ntry: 18, refid: 00000000, correction: 0.000000002, skew: 0.000\ntry: 19, refid: 00000000, correction: 0.000000002, skew: 0.000\ntry: 20, refid: 00000000, correction: 0.000000002, skew: 0.000", "std out_lines": ["try: 1, refid: 00000000, correction: 0.000000000, skew: 0.000", "try: 2, refid: 00000000, correction: 0.000000000, skew: 0.000", "tr y: 3, refid: 00000000, correction: 0.000000000, skew: 0.000", "try: 4, refid: 00000000, correction: 0.000000000, skew: 0.000", "try: 5, refid: 000 00000, correction: 0.000000000, skew: 0.000", "try: 6, refid: 00000000, correction: 0.000000000, skew: 0.000", "try: 7, refid: 00000000, correctio n: 0.000000001, skew: 0.000", "try: 8, refid: 00000000, correction: 0.000000001, skew: 0.000", "try: 9, refid: 00000000, correction: 0.000000001, skew: 0.000", "try: 10, refid: 00000000, correction: 0.000000001, skew: 0.000", "try: 11, refid: 00000000, correction: 0.000000001, skew: 0.000", "try: 12, refid: 00000000, correction: 0.000000001, skew: 0.000", "try: 13, refid: 00000000, correction: 0.000000001, skew: 0.000", "try: 14, refi d: 00000000, correction: 0.000000001, skew: 0.000", "try: 15, refid: 00000000, correction: 0.000000001, skew: 0.000", "try: 16, refid: 00000000, c orrection: 0.000000001, skew: 0.000", "try: 17, refid: 00000000, correction: 0.000000002, skew: 0.000", "try: 18, refid: 00000000, correction: 0.0 00000002, skew: 0.000", "try: 19, refid: 00000000, correction: 0.000000002, skew: 0.000", "try: 20, refid: 00000000, correction: 0.000000002, skew : 0.000"]} Version-Release number of selected component (if applicable): RHOS_TRUNK-16.0-RHEL-8-20200220.n.0 How reproducible: Jenkins job https://rhos-ci-staging-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/edge/job/DFG-edge-deployment-16-rhel-virthost-ipv6-3cont-2comp-2leafs-x-2comp-default/2/console Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
This points to a mis-configuration with unavailable ntp server.
(In reply to Alex Schultz from comment #1) > This points to a mis-configuration with unavailable ntp server. I think, it is configured well. [stack@site-undercloud-0 ~]$ cat overcloud_deploy.sh #!/bin/bash openstack overcloud deploy \ --timeout 240 \ --templates /usr/share/openstack-tripleo-heat-templates \ --stack central \ --libvirt-type kvm \ --ntp-server clock1.rdu2.redhat.com \ ............ [stack@site-undercloud-0 ~]$ cat undercloud.conf [DEFAULT] local_interface = eth0 local_ip = 192.168.24.1/24 undercloud_public_host = 192.168.24.2 undercloud_admin_host = 192.168.24.3 undercloud_ntp_servers=clock.redhat.com,clock2.redhat.com undercloud_nameservers=10.11.5.19 .............. Yuri
No it's not. You should use more than 1 server for the overcloud deployment.
(In reply to Alex Schultz from comment #3) > No it's not. You should use more than 1 server for the overcloud deployment. Well, I have successfully installed Overcloud IPv4 with --ntp-server clock1.rdu2.redhat.com only. Of course, I can check and rerun Overcloud IPv6 with --ntp-server clock1.rdu2.redhat.com,clock2.redhat.com Will update with results.
Just because it worked once doesn't mean it'll always work. We've had issues with the internal ntp servers and it is recommend you use more than one.
(In reply to Alex Schultz from comment #5) > Just because it worked once doesn't mean it'll always work. We've had issues > with the internal ntp servers and it is recommend you use more than one. It always works perfect except the IPv6. Failed again with --ntp-server clock1.rdu2.redhat.com,clock2.redhat.com TASK [Ensure system is NTP time synced] **************************************** Friday 21 February 2020 22:39:26 +0000 (0:00:00.907) 0:03:23.919 ******* fatal: [central-controller0-0]: FAILED! => {"changed": true, "cmd": ["chronyc", "waitsync", "20"], "delta": "0:03:10.198521", "end": "2020-02-21 17:42:36.896896", "msg": "non-zero return code", "rc": 1, "start": "2020-02-21 17:39:26.698375", "stderr": "", "stderr_lines": [], "stdout": "try: 1, refid: 00000000, correction: 0.000000000, skew: 0.000\ntry: 2, refid: 00000000, correction: 0.000000000, skew: 0.000\ntry: 3, refid: 00000000, correction: 0.000000000, skew: 0.000\ntry: 4, refid: 00000000, correction: 0.000000000, skew: 0.000\ntry: 5, refid: 00000000, correction: 0.000000001, skew: 0.000\ntry: 6, refid: 00000000, correction: 0.000000001, skew: 0.000\ntry: 7, refid: 00000000, correction: 0.000000001, skew: 0.000\ntry: 8, refid: 00000000, correction: 0.000000001, skew: 0.000\ntry: 9, refid: 00000000, correction: 0.000000001, skew: 0.000\ntry: 10, refid: 00000000, correction: 0.000000001, skew: 0.000\ntry: 11, refid: 00000000, correction: 0.000000001, skew: 0.000\ntry: 12, refid: 00000000, correction: 0.000000002, skew: 0.000\ntry: 13, refid: 00000000, correction: 0.000000002, skew: 0.000\ntry: 14, refid: 00000000, correction: 0.000000002, skew: 0.000\ntry: 15, refid: 00000000, correction: 0.000000002, skew: 0.000\ntry: 16, refid: 00000000, correction: 0.000000002, skew: 0.000\ntry: 17, refid: 00000000, correction: 0.000000002, skew: 0.000\ntry: 18, refid: 00000000, correction: 0.000000002, skew: 0.000\ntry: 19, refid: 00000000, correction: 0.000000002, skew: 0.000\ntry: 20, refid: 00000000, correction: 0.000000003, skew: 0.000", "stdout_lines": ["try: 1, refid: 00000000, correction: 0.000000000, skew: 0.000", "try: 2, refid: 00000000, correction: 0.000000000, skew: 0.000", "try: 3, refid: 00000000, correction: 0.000000000, skew: 0.000", "try: 4, refid: 00000000, correction: 0.000000000, skew: 0.000", "try: 5, refid: 00000000, correction: 0.000000001, skew: 0.000", "try: 6, refid: 00000000, correction: 0.000000001, skew: 0.000", "try: 7, refid: 00000000, correction: 0.000000001, skew: 0.000", "try: 8, refid: 00000000, correction: 0.000000001, skew: 0.000", "try: 9, refid: 00000000, correction: 0.000000001, skew: 0.000", "try: 10, refid: 00000000, correction: 0.000000001, skew: 0.000", "try: 11, refid: 00000000, correction: 0.000000001, skew: 0.000", "try: 12, refid: 00000000, correction: 0.000000002, skew: 0.000", "try: 13, refid: 00000000, correction: 0.000000002, skew: 0.000", "try: 14, refid: 00000000, correction: 0.000000002, skew: 0.000", "try: 15, refid: 00000000, correction: 0.000000002, skew: 0.000", "try: 16, refid: 00000000, correction: 0.000000002, skew: 0.000", "try: 17, refid: 00000000, correction: 0.000000002, skew: 0.000", "try: 18, refid: 00000000, correction: 0.000000002, skew: 0.000", "try: 19, refid: 00000000, correction: 0.000000002, skew: 0.000", "try: 20, refid: 00000000, correction: 0.000000003, skew: 0.000"]} Something with IPv6. I cannot identify what is the root of issue. I have a running environment where you can get more information. Ping me on IRCC Yuri
Right because those aren't ipv6 capable systems. You'd have to use an IPV6 ntp server. This is an environmental/configuration issue.
I'm not agreed with you. Up to documentation it should be IPv4 address https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.0/html-single/ipv6_networking_for_the_overcloud/index $ openstack overcloud deploy --templates \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation-v6.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-environment-v6.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans-v6.yaml \ -e /home/stack/templates/network-environment.yaml \ --ntp-server pool.ntp.org \ [ADDITIONAL OPTIONS] In OSP 13 it works ok (undercloud) [stack@site-undercloud-0 ~]$ ntpstat synchronised to NTP server (10.11.160.238) at stratum 2 time correct to within 109 ms polling server every 1024 s chrony-3.4-1.el7.x86_64 In OSP 16 does not work. -> chrony-3.5-1.el8.x86_64 If you think it is not related to Openstack, let's escalate to another project. Thank you Yuri
Looks like network configuration is not correct: {\"ip_netmask\": \"169.254.169.254/32\", \"next_hop\": \"\"} ^^ next_hop for metadat is empty. We no longer use metadata over the network, see https://review.opendev.org/665373. So you should drop it from you nic-config file. 2020-02-21 18:44:41,440 p=885 u=mistral | ok: [central-controller0-0] => { "NetworkConfig_result.stderr_lines": [ "+ '[' -n '{\"network_config\": [{\"addresses\": [{\"ip_netmask\": \"192.168.24.42/24\"}], \"dns_servers\": [\"10.11.5.19\", \"10.5.30.160\"], \"name\": \"nic1\", \"routes\": [{\"destination\": \"192.168.34.0/24\", \"nexthop\": \"192.168.24.254\"}, {\"destination\": \"192.168.44.0/24\", \"nexthop\": \"192.168.24.254\"}, {\"ip_netmask\": \"169.254.169.254/32\", \"next_hop\": \"\"}], \"type\": \"interface\", \"use_dhcp\": false}, {\"members\": [{\"mtu\": 1500, \"name\": \"nic2\", \"primary\": true, \"type\": \"interface\"}, {\"addresses\": [{\"ip_netmask\": \"fd00:fd00:fd00:3000::16e/64\"}], \"mtu\": 1500, \"routes\": [{\"destination\": \"fd00:fd00:fd00:3004::/64\", \"nexthop\": \"fd00:fd00:fd00:3000::1\"}, {\"destination\": \"fd00:fd00:fd00:3008::/64\", \"nexthop\": \"fd00:fd00:fd00:3000::1\"}], \"type\": \"vlan\", \"vlan_id\": 1183}, {\"addresses\": [{\"ip_netmask\": \"fd00:fd00:fd00:4000::71/64\"}], \"mtu\": 1500, \"routes\": [{\"destination\": \"fd00:fd00:fd00:4004::/64\", \"nexthop\": \"fd00:fd00:fd00:4000::1\"}, {\"destination\": \"fd00:fd00:fd00:4008::/64\", \"nexthop\": \"fd00:fd00:fd00:4000::1\"}], \"type\": \"vlan\", \"vlan_id\": 1188}, {\"addresses\": [{\"ip_netmask\": \"fd00:fd00:fd00:2000::3f9/64\"}], \"mtu\": 1500, \"routes\": [{\"destination\": \"fd00:fd00:fd00:2004::/64\", \"nexthop\": \"fd00:fd00:fd00:2000::1\"}, {\"destination\": \"fd00:fd00:fd00:2008::/64\", \"nexthop\": \"fd00:fd00:fd00:2000::1\"}], \"type\": \"vlan\", \"vlan_id\": 1185}, {\"addresses\": [{\"ip_netmask\": \"172.19.1.164/24\"}], \"mtu\": 1500, \"routes\": [{\"destination\": \"172.19.2.0/24\", \"nexthop\": \"172.19.1.254\"}, {\"destination\": \"172.19.3.0/24\", \"nexthop\": \"172.19.1.254\"}], \"type\": \"vlan\", \"vlan_id\": 1189}], \"mtu\": 1500, \"name\": \"br-isolated\", \"type\": \"ovs_bridge\", \"use_dhcp\": false}, {\"addresses\": [{\"ip_netmask\": \"2620:52:0:13b8:5054:ff:fe3e:89/64\"}], \"members\": [{\"name\": \"nic3\", \"primary\": true, \"type\": \"interface\"}], \"name\": \"br-ex\", \"routes\": [{\"ip_netmask\": \"::/0\", \"next_hop\": \"2620:52:0:13b8::fe\"}], \"type\": \"ovs_bridge\", \"use_dhcp\": false}]}' ']'", "+ '[' -z '' ']'", "+ trap configure_safe_defaults EXIT", "++ date +%Y-%m-%dT%H:%M:%S", "+ DATETIME=2020-02-21T13:44:17", "+ '[' -f /etc/os-net-config/config.json ']'", "+ mkdir -p /etc/os-net-config", "+ echo '{\"network_config\": [{\"addresses\": [{\"ip_netmask\": \"192.168.24.42/24\"}], \"dns_servers\": [\"10.11.5.19\", \"10.5.30.160\"], \"name\": \"nic1\", \"routes\": [{\"destination\": \"192.168.34.0/24\", \"nexthop\": \"192.168.24.254\"}, {\"destination\": \"192.168.44.0/24\", \"nexthop\": \"192.168.24.254\"}, {\"ip_netmask\": \"169.254.169.254/32\", \"next_hop\": \"\"}], \"type\": \"interface\", \"use_dhcp\": false}, {\"members\": [{\"mtu\": 1500, \"name\": \"nic2\", \"primary\": true, \"type\": \"interface\"}, {\"addresses\": [{\"ip_netmask\": \"fd00:fd00:fd00:3000::16e/64\"}], \"mtu\": 1500, \"routes\": [{\"destination\": \"fd00:fd00:fd00:3004::/64\", \"nexthop\": \"fd00:fd00:fd00:3000::1\"}, {\"destination\": \"fd00:fd00:fd00:3008::/64\", \"nexthop\": \"fd00:fd00:fd00:3000::1\"}], \"type\": \"vlan\", \"vlan_id\": 1183}, {\"addresses\": [{\"ip_netmask\": \"fd00:fd00:fd00:4000::71/64\"}], \"mtu\": 1500, \"routes\": [{\"destination\": \"fd00:fd00:fd00:4004::/64\", \"nexthop\": \"fd00:fd00:fd00:4000::1\"}, {\"destination\": \"fd00:fd00:fd00:4008::/64\", \"nexthop\": \"fd00:fd00:fd00:4000::1\"}], \"type\": \"vlan\", \"vlan_id\": 1188}, {\"addresses\": [{\"ip_netmask\": \"fd00:fd00:fd00:2000::3f9/64\"}], \"mtu\": 1500, \"routes\": [{\"destination\": \"fd00:fd00:fd00:2004::/64\", \"nexthop\": \"fd00:fd00:fd00:2000::1\"}, {\"destination\": \"fd00:fd00:fd00:2008::/64\", \"nexthop\": \"fd00:fd00:fd00:2000::1\"}], \"type\": \"vlan\", \"vlan_id\": 1185}, {\"addresses\": [{\"ip_netmask\": \"172.19.1.164/24\"}], \"mtu\": 1500, \"routes\": [{\"destination\": \"172.19.2.0/24\", \"nexthop\": \"172.19.1.254\"}, {\"destination\": \"172.19.3.0/24\", \"nexthop\": \"172.19.1.254\"}], \"type\": \"vlan\", \"vlan_id\": 1189}], \"mtu\": 1500, \"name\": \"br-isolated\", \"type\": \"ovs_bridge\", \"use_dhcp\": false}, {\"addresses\": [{\"ip_netmask\": \"2620:52:0:13b8:5054:ff:fe3e:89/64\"}], \"members\": [{\"name\": \"nic3\", \"primary\": true, \"type\": \"interface\"}], \"name\": \"br-ex\", \"routes\": [{\"ip_netmask\": \"::/0\", \"next_hop\": \"2620:52:0:13b8::fe\"}], \"type\": \"ovs_bridge\", \"use_dhcp\": false}]}'", "++ type -t network_config_hook", "+ '[' '' = function ']'", "+ sed -i 's/: \"bridge_name/: \"br-ex/' /etc/os-net-config/config.json", "+ sed -i s/interface_name/nic1/ /etc/os-net-config/config.json", "+ set +e", "+ os-net-config -c /etc/os-net-config/config.json -v --detailed-exit-codes", "[2020/02/21 01:44:17 PM] [INFO] Using config file at: /etc/os-net-config/config.json", "[2020/02/21 01:44:17 PM] [INFO] Ifcfg net config provider created.", "[2020/02/21 01:44:17 PM] [INFO] Not using any mapping file.", "[2020/02/21 01:44:18 PM] [WARNING] Config file failed schema validation at network_config/0:", " {'addresses': [{'ip_netmask': '192.168.24.42/24'}], 'dns_servers': ['10.11.5.19', '10.5.30.160'], 'name': 'nic1', 'routes': [{'destination': '192.168.34.0/24', 'nexthop': '192.168.24.254'}, {'destination': '192.168.44.0/24', 'nexthop': '192.168.24.254'}, {'ip_netmask': '169.254.169.254/32', 'next_hop': ''}], 'type': 'interface', 'use_dhcp': False, 'nic_mapping': None, 'persist_mapping': False} is not valid under any of the given schemas", " Sub-schemas tested and not matching:", " - items/oneOf/interface/routes/items/oneOf: {'ip_netmask': '169.254.169.254/32', 'next_hop': ''} is not valid under any of the given schemas", " -- items/oneOf/interface/routes/items/oneOf/0/next_hop/oneOf: '' is not valid under any of the given schemas", " --- items/oneOf/interface/routes/items/oneOf/0/next_hop/oneOf/ip_address_string/oneOf: '' is not valid under any of the given schemas", " ---- items/oneOf/interface/routes/items/oneOf/0/next_hop/oneOf/ip_address_string/oneOf/ipv4_address_string/pattern: '' does not match '^((25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?).){3}(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)$'", " ---- items/oneOf/interface/routes/items/oneOf/0/next_hop/oneOf/ip_address_string/oneOf/ipv6_address_string/pattern: '' does not match '^(([0-9a-fA-F]{1,4}:){7,7}[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,7}:|([0-9a-fA-F]{1,4}:){1,6}:[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,5}(:[0-9a-fA-F]{1,4}){1,2}|([0-9a-fA-F]{1,4}:){1,4}(:[0-9a-fA-F]{1,4}){1,3}|([0-9a-fA-F]{1,4}:){1,3}(:[0-9a-fA-F]{1,4}){1,4}|([0-9a-fA-F]{1,4}:){1,2}(:[0-9a-fA-F]{1,4}){1,5}|[0-9a-fA-F]{1,4}:((:[0-9a-fA-F]{1,4}){1,6})|:((:[0-9a-fA-F]{1,4}){1,7}|:))$'", " --- items/oneOf/interface/routes/items/oneOf/0/next_hop/oneOf/param/oneOf: '' is not valid under any of the given schemas", " ---- items/oneOf/interface/routes/items/oneOf/0/next_hop/oneOf/param/oneOf/0/type: '' is not of type 'object'", " ---- items/oneOf/interface/routes/items/oneOf/0/next_hop/oneOf/param/oneOf/1/type: '' is not of type 'object'", " -- items/oneOf/interface/routes/items/oneOf/1/additionalProperties: Additional properties are not allowed ('next_hop', 'ip_netmask' were unexpected)",