Created attachment 1478250 [details] /var/lib/mistral/xyz/ansible.log Description of problem: Deployments of OSP14 ipv6 jobs fail on following error during "NetworkDeployment" mistral task, that appears on controller nodes: ... [WARNING] Config file failed schema validation at network_config/0:", " {'persist_mapping': False, 'addresses': [{'ip_netmask': '192.168.24.12/'}], 'nic_mapping': None, 'routes': [{'ip_netmask': '169.254.169.254/32', 'next_hop': '192.168.24.1'}, {'default': True, 'next_hop': '192.168.24.1'}], 'use_dhcp': False, 'type': 'interface', 'name': 'nic1'} is not valid under any of the given schemas", " Sub-schemas tested and not matching:", " - items/oneOf/interface/addresses/items/ip_netmask/oneOf: '192.168.24.12/' is not valid under any of the given schemas", " -- items/oneOf/interface/addresses/items/ip_netmask/oneOf/ip_cidr_string/oneOf: '192.168.24.12/' is not valid under any of the given schemas", " --- items/oneOf/interface/addresses/items/ip_netmask/oneOf/ip_cidr_string/oneOf/ipv4_cidr_string/pattern: '192.168.24.12/' does not match '^((25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?).){3}(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)/(3[0-2]|[1-2][0-9]|[0-9])$'", " --- items/oneOf/interface/addresses/items/ip_netmask/oneOf/ip_cidr_string/oneOf/ipv6_cidr_string/pattern: '192.168.24.12/' does not match '^(([0-9a-fA-F]{1,4}:){7,7}[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,7}:|([0-9a-fA-F]{1,4}:){1,6}:[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,5}(:[0-9a-fA-F]{1,4}){1,2}|([0-9a-fA-F]{1,4}:){1,4}(:[0-9a-fA-F]{1,4}){1,3}|([0-9a-fA-F]{1,4}:){1,3}(:[0-9a-fA-F]{1,4}){1,4}|([0-9a-fA-F]{1,4}:){1,2}(:[0-9a-fA-F]{1,4}){1,5}|[0-9a-fA-F]{1,4}:((:[0-9a-fA-F]{1,4}){1,6})|:((:[0-9a-fA-F]{1,4}){1,7}|:))/(12[0-8]|1[0-1][0-9]|[1-9][0-9]|[0-9])$'", " -- items/oneOf/interface/addresses/items/ip_netmask/oneOf/param/oneOf: '192.168.24.12/' is not valid under any of the given schemas", " --- items/oneOf/interface/addresses/items/ip_netmask/oneOf/param/oneOf/0/type: '192.168.24.12/' is not of type 'object'", " --- items/oneOf/interface/addresses/items/ip_netmask/oneOf/param/oneOf/1/type: '192.168.24.12/' is not of type 'object'", " -- items/oneOf/interface/addresses/items/ip_netmask/oneOf/2/type: '192.168.24.12/' is not of type 'object'", "Traceback (most recent call last):", " File \"/bin/os-net-config\", line 10, in <module>", " sys.exit(main())", " File \"/usr/lib/python2.7/site-packages/os_net_config/cli.py\", line 259, in main", " obj = objects.object_from_json(iface_json)", " File \"/usr/lib/python2.7/site-packages/os_net_config/objects.py\", line 43, in object_from_json", " return Interface.from_json(json)", " File \"/usr/lib/python2.7/site-packages/os_net_config/objects.py\", line 415, in from_json", " opts = _BaseOpts.base_opts_from_json(json)", " File \"/usr/lib/python2.7/site-packages/os_net_config/objects.py\", line 364, in base_opts_from_json", " addresses.append(Address.from_json(address))", " File \"/usr/lib/python2.7/site-packages/os_net_config/objects.py\", line 269, in from_json", " return Address(ip_netmask)", " File \"/usr/lib/python2.7/site-packages/os_net_config/objects.py\", line 260, in __init__", " ip_nw = netaddr.IPNetwork(self.ip_netmask)", " File \"/usr/lib/python2.7/site-packages/netaddr/ip/__init__.py\", line 938, in __init__", " raise AddrFormatError('invalid IPNetwork %s' % addr)", "netaddr.core.AddrFormatError: invalid IPNetwork 192.168.24.6/", "+ RETVAL=1", "+ set -e", "+ [[ 1 == 2 ]]", "+ [[ 1 != 0 ]]", "+ echo 'ERROR: os-net-config configuration failed.'", "ERROR: os-net-config configuration failed.", "+ exit 1", ... 2018-08-22 09:20:05,510 p=820 u=mistral | NO MORE HOSTS LEFT ************************************************************* 2018-08-22 09:20:05,511 p=820 u=mistral | PLAY RECAP ********************************************************************* 2018-08-22 09:20:05,511 p=820 u=mistral | compute-0 : ok=5 changed=2 unreachable=0 failed=0 2018-08-22 09:20:05,511 p=820 u=mistral | controller-0 : ok=18 changed=4 unreachable=0 failed=1 2018-08-22 09:20:05,512 p=820 u=mistral | controller-1 : ok=18 changed=4 unreachable=0 failed=1 2018-08-22 09:20:05,512 p=820 u=mistral | controller-2 : ok=18 changed=4 unreachable=0 failed=1 2018-08-22 09:20:05,512 p=820 u=mistral | undercloud : ok=2 changed=0 unreachable=0 failed=0 2018-08-22 09:20:05,513 p=820 u=mistral | Wednesday 22 August 2018 09:20:05 -0400 (0:00:00.129) 0:00:32.421 ****** 2018-08-22 09:20:05,513 p=820 u=mistral | =============================================================================== Version-Release number of selected component (if applicable): OSP14, containerized UC, puddle 2018-08-21.2 Steps to Reproduce: 1. Deploy OSP14 1uc,1contr,1compute, ipv6, no ceph node, ssl or w/o ssl using InfraRed Additional info (UC): os-net-config.noarch 9.1.1-0.20180730115318.e128373.el7ost
Note the net_ip_map has the addresses with missing cidr fir the ctlplane subnet, here is for controller-2 in /var/lib/mistral/overcloud/Controller/controller-2/ControllerDeployment: "net_ip_map": { "tenant": "172.17.2.24", "management": "192.168.24.12", "tenant_uri": "172.17.2.24", "ctlplane_uri": "192.168.24.12", "management_uri": "192.168.24.12", "management_subnet": "192.168.24.12/", "storage": "fd00:fd00:fd00:3000::15", "internal_api_subnet": "fd00:fd00:fd00:2000::19/64", "storage_subnet": "fd00:fd00:fd00:3000::15/64", "external_subnet": "2620:52:0:13b8:5054:ff:fe3e:7/64", "ctlplane": "192.168.24.12", "storage_mgmt_subnet": "fd00:fd00:fd00:4000::11/64", "external": "2620:52:0:13b8:5054:ff:fe3e:7", "ctlplane_subnet": "192.168.24.12/", <=== Note also, it appears that change https://review.openstack.org/#/c/588122/ is not included in this puddle. Since deployed servers are not being used this should not affect anything. So the cidr in net_ip_map is set up here (https://review.openstack.org/#/c/579579/): NetIpMap: type: OS::TripleO::Network::Ports::NetIpMap properties: ControlPlaneIp: {get_attr: [{{server_resource_name}}, networks, ctlplane, 0]} ControlPlaneSubnetCidr: if: - ctlplane_subnet_cidr_set - {get_param: ControlPlaneSubnetCidr} - {str_split: ['/', {get_attr: [{{server_resource_name}}, addresses, ctlplane, 0, subnets, 0, cidr]}, 1]} This test uses virt/network/network-environment-v6.yaml which does not define ControlPlaneSubnetCidr. So its possible that there is an issue if ControlPlaneSubnetCidr is not explicitly defined. The IPv4 version of network-environment.yaml - virt/network/network-environment.yaml, does define ControlPlaneSubnetCidr: "24" I believe the IPv4 test passes.
I was able to deploy ipv6 overcloud using InfraRed with workaround ControlPlaneSubnetCidr: "24" added to /home/stack/virt/network/network-environment-v6.yaml (https://review.gerrithub.io/c/redhat-openstack/infrared/+/423696) manually before OC deployment itself. Downstream ipv4 deployments use different network-environment.yaml that has this variable/value already included in their config: https://github.com/redhat-openstack/infrared/blob/master/plugins/tripleo-overcloud/vars/deployment/files/virt/network/network-environment.yaml.j2#L41 .
Thanks Filip, that confirms that there is an issue if ControlPlaneSubnetCidr is not explicitly defined.
This was missing patch - https://review.openstack.org/#/c/568960/
Filip, Yes, it looks like openstack-heat-11.0.0-0.20180822213000.207498d.el7ost was built for OSP-14, see https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=760312. In puddle 2018-09-06.1 it looks like openstack-heat-11.0.1-0.20180901130821.680a515.el7ost.src is being used which should have this fix. As for why puddle 2018-08-21.2 is using openstack-heat-engine-12.0.0-0 (I've also verified that puddle 2018-08-23.3 uses openstack-heat-12.0.0-0.20180604085325.7d878a8.el7ost.src) and later puddle use 11.0 I'm not sure. I've added a NeedInfo for Lon to see if he can help clear this up. To test this I'd recommend picking up a later puddle that has this fix to ensure the package is picked up in all places.
11.0 is the correct version. 12.0 was a versioning mistake Use: rpm -Uvh --oldpackage ...
The first version I can verify on is openstack-heat-11.0.1-0.20180901130821.680a515.el7ost family, whose packages are present throughout all openstack-heat* containers in deployment and also on undercloud node in puddle 2018-09-06.1. Deployment passes also when ControlPlaneSubnetCidr is omitted. ------ Since puddle 2018-08-21.2 contains openstack-heat-12.0.0-0.20180604085325.7d878a8.el7ost on UC and inside containers, I was not able to install version of currently fixed package (11.0.0-0.20180822213000.207498) container-wise, just UC-wise, because newer openstack-heat-11.0.0-0.20180822213000.207498.el7ost is pulling more dependencies (python-pecan, ...) that have system-wide dependencies not compatible with what we have inside heat* containers (openstack-heat-12.0.0-0.20180604085325.7d878a8.el7ost). Also manual and forced installation into containers on such version will pass and services come up, but results into issues anyway: forall_heat-containers $ rpm -Uvh --oldpackage --force --nodeps *11.0.0-0.20180822213000.207498*.rpm resources.TenantNetwork: Property error: resources.TenantSubnet.properties.allocation_pools[0].start: "172.17.2.10" does not validate ip_addr (constraint not found) # during OC deploy Moving the "Fixed in" version to openstack-heat-11.0.1-0.20180901130821.680a515.el7ost then.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:0045