Description of problem: OC deployment of OSP17 fails on missing THT file deployed_vip_external.yaml: WARNING tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] jinja2 rendering roles Controller,Compute,BlockStorage,ObjectStorage,CephStorage[00m WARNING tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] rendering j2 template to file: /home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/puppet/controller-role.yaml[00m WARNING tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] rendering j2 template to file: /home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/puppet/compute-role.yaml[00m WARNING tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] rendering j2 template to file: /home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/puppet/blockstorage-role.yaml[00m WARNING tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] rendering j2 template to file: /home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/puppet/objectstorage-role.yaml[00m WARNING tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] rendering j2 template to file: /home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/puppet/cephstorage-role.yaml[00m ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] Exception occured while running the command: heatclient.exc.CommandError: Could not fetch contents for file:///home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/network/ports/deployed_vip_external.yaml ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud Traceback (most recent call last): ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib64/python3.6/urllib/request.py", line 1505, in open_local_file ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud stats = os.stat(localfile) ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud FileNotFoundError: [Errno 2] No such file or directory: '/usr/share/openstack-tripleo-heat-templates/network/ports/deployed_vip_external.yaml' ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud During handling of the above exception, another exception occurred: ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud Traceback (most recent call last): ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/heatclient/common/utils.py", line 415, in read_url_content ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud content = request.urlopen(url).read() ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib64/python3.6/urllib/request.py", line 223, in urlopen ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud return opener.open(url, data, timeout) ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib64/python3.6/urllib/request.py", line 526, in open ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud response = self._open(req, data) ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib64/python3.6/urllib/request.py", line 544, in _open ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud '_open', req) ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib64/python3.6/urllib/request.py", line 504, in _call_chain ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud result = func(*args) ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib64/python3.6/urllib/request.py", line 1483, in file_open ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud return self.open_local_file(req) ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib64/python3.6/urllib/request.py", line 1522, in open_local_file ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud raise URLError(exp) ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud urllib.error.URLError: <urlopen error [Errno 2] No such file or directory: '/usr/share/openstack-tripleo-heat-templates/network/ports/deployed_vip_external.yaml' ... ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud [00m ERROR openstack [-] Could not fetch contents for file:///home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/network/ports/deployed_vip_external.yaml: heatclient.exc.CommandError: Could not fetch contents for file:///home/stack/overcloud-deploy/overcloud/tripleo-heat-templates/network/ports/deployed_vip_external.yaml[00m INFO osc_lib.shell [-] END return value: 1[00m Version-Release number of selected component (if applicable): RHOS-17.0-RHEL-8-20211105.n.0 Additional info: openstack-tripleo-heat-templates-14.3.1-0.20211030221907.6b66b70.el8ost.noarch
The deployed_vip_<network>.yaml is rendered based on provided network data. It looks like it's not being correctly passed 2021-11-05 03:14:41.639 108580 WARNING tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] jinja2 rendering network template deployed_port.network.j2.yaml[00m 2021-11-05 03:14:41.639 108580 WARNING tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] jinja2 rendering networks[00m 2021-11-05 03:14:41.640 108580 WARNING tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] jinja2 rendering network template deployed_vip_port.network.j2.yaml[00m 2021-11-05 03:14:41.640 108580 WARNING tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] jinja2 rendering networks[00m 2021-11-05 03:14:41.640 108580 WARNING tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] jinja2 rendering network template external_resource_port.network.j2.yaml[00m 2021-11-05 03:14:41.640 108580 WARNING tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] jinja2 rendering networks[00m 2021-11-05 03:14:41.640 108580 WARNING tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] jinja2 rendering network template external_resource_port_v6.network.j2.yaml[00m 2021-11-05 03:14:41.640 108580 WARNING tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] jinja2 rendering networks[00m When comparing an upstream job, we see 2021-11-01 16:31:06 | 2021-11-01 16:31:06.749 321894 WARNING tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] jinja2 rendering network template deployed_vip_port.network.j2.yaml[00m 2021-11-01 16:31:06 | 2021-11-01 16:31:06.749 321894 WARNING tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] jinja2 rendering networks External,InternalApi,Storage,StorageMgmt,Tenant[00m 2021-11-01 16:31:06 | 2021-11-01 16:31:06.749 321894 WARNING tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] rendering j2 template to file: /home/zuul/overcloud-deploy/overcloud/tripleo-heat-templates/network/ports/deployed_vip_external.yaml[00m 2021-11-01 16:31:06 | 2021-11-01 16:31:06.749 321894 WARNING tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] rendering j2 template to file: /home/zuul/overcloud-deploy/overcloud/tripleo-heat-templates/network/ports/deployed_vip_internal_api.yaml[00m 2021-11-01 16:31:06 | 2021-11-01 16:31:06.749 321894 WARNING tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] rendering j2 template to file: /home/zuul/overcloud-deploy/overcloud/tripleo-heat-templates/network/ports/deployed_vip_storage.yaml[00m 2021-11-01 16:31:06 | 2021-11-01 16:31:06.750 321894 WARNING tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] rendering j2 template to file: /home/zuul/overcloud-deploy/overcloud/tripleo-heat-templates/network/ports/deployed_vip_storage_mgmt.yaml[00m 2021-11-01 16:31:06 | 2021-11-01 16:31:06.750 321894 WARNING tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] rendering j2 template to file: /home/zuul/overcloud-deploy/overcloud/tripleo-heat-templates/network/ports/deployed_vip_tenant.yaml[00m 2021-11-01 16:31:06 | 2021-11-01 16:31:06.750 321894 WARNING tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] jinja2 rendering normal template net_vip_map_external.j2.yaml[00m The deployment command is missing --networks-file so it's using the default which is empty (THT/network_data_default.yaml) https://github.com/openstack/tripleo-heat-templates/blob/stable/wallaby/network_data_default.yaml You need to specify a --networks-file with the networks correctly defined. Upstream we use: https://github.com/openstack/tripleo-heat-templates/blob/stable/wallaby/ci/network_data.yaml This appears to be a job misconfiguration related to the network-v2 work.
Need to make sure this is documented as part of the network-v2 changes needed for OSP17
Alex's analysis in comment #2 is correct. With network v2 the default is no networks, replacing the previous behavior where the all the default network-isolation networks was semi enabled. Pre network-v2 the networks would all fall back to the ip address of the ctlplane if not including a network-isolation environment file. With network-v2 the networks defined in --networks-file controls what networks to use, the default is no-networks - i.e only the ctlplane network. The job need to be correctly re-configured for network-v2. I have changed component to documentation.
Ok, thanks for the explanation. For the record, my understanding is this requires current way of overcloud deployment: ... --deployed-server -e /home/stack/templates/overcloud-vip-deployed.yaml -e /home/stack/templates/overcloud-networks-deployed.yaml -e /home/stack/templates/overcloud-baremetal-deployed.yaml ... to be changed to ... --deployed-server -e /home/stack/templates/overcloud-vip-deployed.yaml -e /home/stack/templates/overcloud-networks-deployed.yaml -e /home/stack/templates/overcloud-baremetal-deployed.yaml --networks-file /home/stack/virt/network/network_data_v2.yaml ... (in addition the network_data_v2.yaml file is still taken as input for the "overcloud network provision" step being done earlier in the deployment) --- Testing IR change here: https://review.gerrithub.io/c/redhat-openstack/infrared/+/527052 - though with that change I receive another OC deployment error further in the deployment, which I suspect might be related to this change as the OC endpoint IP doesn't seem to be set? Or could it be a different issue that needs a new BZ? ... PLAY RECAP ********************************************************************* localhost : ok=2 changed=1 unreachable=0 failed=0 skipped=1 rescued=0 ignored=0 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Total Tasks: 3 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Elapsed Time: 0:00:03.757627 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | UUID | Info | Host | Task Name | Run Time | 525400d0-3da6-0c14-1c48-00000000000a | SUMMARY | localhost | Fetch stack data | 2.97s | 525400d0-3da6-0c14-1c48-00000000000b | SUMMARY | localhost | Create horizon url temp file | 0.68s | 525400d0-3da6-0c14-1c48-000000000008 | SUMMARY | localhost | Check for required inputs | 0.04s | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ End Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ INFO tripleoclient.utils.utils [-] Temporary directory [ /tmp/tripleodgsygeri ] cleaned up INFO tripleoclient.utils.utils [-] Ansible execution success. playbook: /usr/share/ansible/tripleo-playbooks/cli-undercloud-get-horizon-url.yaml WARNING tripleoclient.utils.safe_write [-] The output file /home/stack/overcloud-deploy/overcloud/overcloud-deployment_status.yaml will be overriden ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] Exception creating overcloud export.: UnboundLocalError: local variable 'config_download_dir' referenced before assignment ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] local variable 'config_download_dir' referenced before assignment: UnboundLocalError: local variable 'config_download_dir' referenced before assignm ent Overcloud Endpoint: http://:5000 Overcloud Horizon Dashboard URL: http://:80/dashboard Overcloud rc file: None and None Overcloud Deployed with error INFO tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] Stopping ephemeral heat. INFO tripleoclient.heat_launcher [-] Killing pod: ephemeral-heat b9121ad88dc825e5a2614c74c20bfc4ce INFO tripleoclient.heat_launcher [-] Killed pod: ephemeral-heat INFO tripleoclient.heat_launcher [-] Starting back up of heat db INFO tripleoclient.heat_launcher [-] Created tarfile /home/stack/overcloud-deploy/overcloud/heat-launcher/heat-db.sql-1636552942.4944806.tar.bzip2 INFO tripleoclient.heat_launcher [-] Deleting /home/stack/overcloud-deploy/overcloud/heat-launcher/heat-db.sql INFO tripleoclient.heat_launcher [-] Removing pod: ephemeral-heat b9121ad88dc825e5a2614c74c20bfc4ce INFO tripleoclient.heat_launcher [-] Created tarfile /home/stack/overcloud-deploy/overcloud/heat-launcher/log/heat-1636552942.4944806.log-1636552942.4944806.tar.bzip2 INFO tripleoclient.heat_launcher [-] Deleting /home/stack/overcloud-deploy/overcloud/heat-launcher/log/heat-1636552942.4944806.log ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] Exception occured while running the command: TypeError: inet_pton() argument 2 must be str, not None ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud Traceback (most recent call last): ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 34, in run ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud super(Command, self).run(parsed_args) ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 39, in run ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud return super(Command, self).run(parsed_args) ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/cliff/command.py", line 185, in run ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud return_code = self.take_action(parsed_args) or 0 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1427, in take_action ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud working_dir=self.working_dir) ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__ ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud self.force_reraise() ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud raise self.value ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1328, in take_action ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud stack, rc_params, parsed_args.no_proxy) ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/deployment.py", line 40, in create_overcloudrc ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud rc_params['region']) ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleo_common/utils/overcloudrc.py", line 87, in _create_overcloudrc ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud no_proxy_list = sorted(list(set(no_proxy_list))) ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleo_common/utils/common.py", line 24, in bracket_ipv6 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud socket.inet_pton(socket.AF_INET6, address) ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud TypeError: inet_pton() argument 2 must be str, not None ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud ERROR openstack [-] inet_pton() argument 2 must be str, not None INFO osc_lib.shell [-] END return value: 1
It looks like we're still missing one of the files here. Can we review the documentation here and ensure we have all of the files passed to the overcloud deploy command? https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/provisioning/baremetal_provision.html#deploying-the-overcloud So you should have: -e /usr/share/openstack-tripleo-heat-templates/environments/deployed-server-environment.yaml \ -e ~/overcloud-networks-deployed.yaml \ -e ~/templates/vips-deployed-environment.yaml \ -e ~/overcloud-baremetal-deployed.yaml \ --deployed-server \ --disable-validations \ Looks like one of them is missing there. Here's practical examples of what those file contents should look similar to: https://github.com/r3d3mpt10n/tripleo-home/blob/main/baremetal-deployed.yaml https://github.com/r3d3mpt10n/tripleo-home/blob/main/networks-deployed.yaml https://github.com/r3d3mpt10n/tripleo-home/blob/main/vips-deployed.yaml Are we able to confirm the overcloud_deploy command that was used for this environment?
I fixed the problem mentioned at https://bugzilla.redhat.com/show_bug.cgi?id=2020590#c5 by https://review.gerrithub.io/c/redhat-openstack/infrared/+/527510 (on top of https://review.gerrithub.io/c/redhat-openstack/infrared/+/527052) - by comparison with https://github.com/openstack/tripleo-heat-templates/blob/stable/wallaby/ci/network_data.yaml mentioned by Alex I noticed we were missing External VIP being set in our network_data_v2.yaml - that explained the missing value in overcloudrc. For the record, the current OC deployment command layout used is: openstack overcloud deploy \ --timeout 100 \ --templates /usr/share/openstack-tripleo-heat-templates \ --stack overcloud \ --libvirt-type kvm \ --ntp-server ... \ --deployed-server -e /home/stack/templates/overcloud-vip-deployed.yaml -e /home/stack/templates/overcloud-networks-deployed.yaml -e /home/stack/templates/overcloud-baremetal-deployed.yaml \ --networks-file /home/stack/virt/network/network_data_v2.yaml \ --disable-protected-resource-types \ -e /home/stack/virt/internal.yaml \ -e /home/stack/virt/network/network-environment_v2.yaml \ -e /home/stack/virt/hostnames.yml \ -e /usr/share/openstack-tripleo-heat-templates/environments/cephadm/cephadm.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovn-ha.yaml \ -e /home/stack/virt/debug.yaml \ -e /home/stack/virt/ntp_pool.yaml \ -e /home/stack/virt/nodes_data.yaml \ -e ~/containers-prepare-parameter.yaml \ -e /home/stack/virt/docker-images.yaml \ --log-file overcloud_deployment_78.log Thanks for the help, this is fixed in InfraRed now and deployment passes phase1. The remaining action item seems to be a proper documentation adjustment.
Hi, The RHOSP 17.0 and 17.1 Director Installation and Usage guides have been updated. Customers can see these changes here: https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/17.0/html/director_installation_and_usage/index https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/17.1/html/installing_and_managing_red_hat_openstack_platform_with_director/index --Greg