Hide Forgot
Description of problem: Overcloud deployment using external loadbalancer cluster fails with: overcloud.AllNodesDeploySteps.ControllerDeployment_Step2.0: resource_type: OS::Heat::StructuredDeployment physical_resource_id: 6a0471af-8cf0-4382-be11-fbe4c33c75dc status: CREATE_FAILED status_reason: | Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1 deploy_stdout: | Notice: hiera(): Cannot load backend module_data: cannot load such file -- hiera/backend/module_data_backend Notice: Scope(Class[Tripleo::Firewall::Post]): At this stage, all network traffic is blocked. Notice: Compiled catalog for overcloud-controller-0.localdomain in environment production in 8.20 seconds deploy_stderr: | ... Warning: ModuleLoader: module 'timezone' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules (file & line not available) Warning: Unknown variable: 'osd_caps'. at /etc/puppet/modules/ceph/manifests/key.pp:101:25 Warning: Unknown variable: 'mds_caps'. at /etc/puppet/modules/ceph/manifests/key.pp:101:36 Warning: Unknown variable: 'mds_caps'. at /etc/puppet/modules/ceph/manifests/key.pp:101:36 Warning: ModuleLoader: module 'openstacklib' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules (file & line not available) Warning: Could not find resource 'Mysql_user[nova_api@cell0]' in parameter 'require' (at /etc/puppet/modules/openstacklib/manifests/db/mysql/host_access.pp:59) Error: Could not find dependency Mysql_user[nova_api@cell0] for Mysql_grant[nova_api@cell0/nova_api_cell0.*] at /etc/puppet/modules/openstacklib/manifests/db/mysql/host_access.pp:56 (truncated, view all with --long) Version-Release number of selected component (if applicable): openstack-tripleo-heat-templates-6.0.0-0.20170120220933.36d493c.el7ost.noarch How reproducible: 100% Steps to Reproduce: 1. Deploy overcloud with custom roles and using an external load balancer: source ~/stackrc export THT=/usr/share/openstack-tripleo-heat-templates/ openstack overcloud deploy --templates $THT \ -r ~/openstack_deployment/roles/roles_data.yaml \ -e $THT/environments/puppet-pacemaker.yaml \ -e $THT/environments/network-isolation.yaml \ -e $THT/environments/network-management.yaml \ -e $THT/environments/storage-environment.yaml \ -e $THT/environments/external-loadbalancer-vip.yaml \ -e ~/openstack_deployment/environments/nodes.yaml \ -e ~/openstack_deployment/environments/network-environment.yaml \ -e ~/openstack_deployment/environments/disk-layout.yaml \ -e ~/openstack_deployment/environments/neutron-settings.yaml \ -e ~/openstack_deployment/environments/external-lb.yaml \ --log-file overcloud_deployment.log &> overcloud_install.log Actual results: Deployment fails. Expected results: Deployment passes. Additional info: Environment files: http://paste.openstack.org/show/596412/
This bug should not be here anymore, we removed the nova_cell thing in puppet-nova (code was bad). It's a build problem I think, where puppet-tripleo needs an update.
(In reply to Emilien Macchi from comment #1) > This bug should not be here anymore, we removed the nova_cell thing in > puppet-nova (code was bad). It's a build problem I think, where > puppet-tripleo needs an update. Can you link to the patch in puppet-nova and move to POST?
or puppet-tripleo
Marius, can you check this one is good? Thanks!
(In reply to Emilien Macchi from comment #4) > Marius, can you check this one is good? > > Thanks! No, the error is till there. Please note that the Nova control plane services run on a different role than controller so this could be related. This is the roles_data.yaml: http://paste.openstack.org/show/601095/ Attaching the os-collect-config log on controller which shows the error.
Created attachment 1259069 [details] os-collect-config log
Looking at the environment that Marius provided, MySQLClient service is missing, so Puppet fails to create the mysql users on ServiceApi role: Error: Could not find dependency Mysql_user[nova@cell0] for Mysql_grant[nova@cell0/nova_cell0.*] Marking this bug as invalid for now but feel free to re-open it if there is something else.
The problem is related to your environment, you need to comment: OS::TripleO::Network::Ports::NetVipMap: /usr/share/openstack-tripleo-heat-templates/network/ports/net_vip_map_external.yaml OS::TripleO::Network::Ports::ExternalVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml OS::TripleO::Network::Ports::InternalApiVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml OS::TripleO::Network::Ports::StorageVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml OS::TripleO::Network::Ports::StorageMgmtVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml OS::TripleO::Network::Ports::RedisVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/from_service.yaml We don't want these noop files otherwise we'll use the Control Plane VIP which is... empty. Please try again without these lines and tell me if it helped. Also, I'm wondering if we need a release note about that (with good documentation for our users.
(In reply to Emilien Macchi from comment #8) > The problem is related to your environment, you need to comment: > > OS::TripleO::Network::Ports::NetVipMap: > /usr/share/openstack-tripleo-heat-templates/network/ports/ > net_vip_map_external.yaml > OS::TripleO::Network::Ports::ExternalVipPort: > /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml > OS::TripleO::Network::Ports::InternalApiVipPort: > /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml > OS::TripleO::Network::Ports::StorageVipPort: > /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml > OS::TripleO::Network::Ports::StorageMgmtVipPort: > /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml > OS::TripleO::Network::Ports::RedisVipPort: > /usr/share/openstack-tripleo-heat-templates/network/ports/from_service.yaml > > We don't want these noop files otherwise we'll use the Control Plane VIP > which is... empty. > > Please try again without these lines and tell me if it helped. > > Also, I'm wondering if we need a release note about that (with good > documentation for our users. Thanks, it looks like this is a change introduced in Ocata by https://github.com/openstack/tripleo-heat-templates/commit/a3f03eb307797ac5eef1251b9252e642db326e07 I'll test it tomorrow in the morning and get back with my results. We probably need to add this to the docs.
Deployment completed successfully with the following environment file. I am switching this BZ to docs so we make sure we cover the changes in the product documentation. resource_registry: OS::TripleO::Controller::Ports::ExternalPort: /usr/share/openstack-tripleo-heat-templates/network/ports/external_from_pool.yaml OS::TripleO::Controller::Ports::InternalApiPort: /usr/share/openstack-tripleo-heat-templates/network/ports/internal_api_from_pool.yaml OS::TripleO::Controller::Ports::StoragePort: /usr/share/openstack-tripleo-heat-templates/network/ports/storage_from_pool.yaml OS::TripleO::Controller::Ports::StorageMgmtPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml OS::TripleO::Controller::Ports::TenantPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml OS::TripleO::Controller::Ports::ManagementPort: /usr/share/openstack-tripleo-heat-templates/network/ports/management_from_pool.yaml OS::TripleO::ServiceApi::Ports::ExternalPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml OS::TripleO::ServiceApi::Ports::InternalApiPort: /usr/share/openstack-tripleo-heat-templates/network/ports/internal_api_from_pool.yaml OS::TripleO::ServiceApi::Ports::StoragePort: /usr/share/openstack-tripleo-heat-templates/network/ports/storage_from_pool.yaml OS::TripleO::ServiceApi::Ports::StorageMgmtPort: /usr/share/openstack-tripleo-heat-templates/network/ports/storage_mgmt_from_pool.yaml OS::TripleO::ServiceApi::Ports::TenantPort: /usr/share/openstack-tripleo-heat-templates/network/ports/tenant_from_pool.yaml OS::TripleO::ServiceApi::Ports::ManagementPort: /usr/share/openstack-tripleo-heat-templates/network/ports/management_from_pool.yaml parameter_defaults: ServiceNetMap: KeystoneAdminApiNetwork: internal_api # allows undercloud to config endpoints RedisPassword: 'examplepassword' ControlFixedIPs: [{'ip_address':'192.168.0.120'}] PublicVirtualFixedIPs: [{'ip_address':'172.16.18.120'}] InternalApiVirtualFixedIPs: [{'ip_address':'10.0.0.120'}] StorageVirtualFixedIPs: [{'ip_address':'10.0.0.220'}] StorageMgmtVirtualFixedIPs: [{'ip_address':'10.0.1.120'}] RedisVirtualFixedIPs: [{'ip_address':'10.0.0.119'}] ControllerIPs: external: - 172.16.18.121 - 172.16.18.122 - 172.16.18.123 internal_api: - 10.0.0.121 - 10.0.0.122 - 10.0.0.123 storage: - 10.0.0.221 - 10.0.0.222 - 10.0.0.223 storage_mgmt: - 10.0.1.101 - 10.0.1.102 - 10.0.1.103 tenant: - 10.0.1.201 - 10.0.1.202 - 10.0.1.203 management: - 172.16.17.201 - 172.16.17.202 - 172.16.17.203 ServiceApiIPs: internal_api: - 10.0.0.124 - 10.0.0.125 - 10.0.0.126 storage: - 10.0.0.224 - 10.0.0.225 - 10.0.0.226 storage_mgmt: - 10.0.1.104 - 10.0.1.105 - 10.0.1.106 tenant: - 10.0.1.204 - 10.0.1.205 - 10.0.1.206 management: - 172.16.17.204 - 172.16.17.205 - 172.16.17.206
Hey Dan, Can you maybe help on this one? I've added some DocText to help, let me know if you need more context. Thanks!
Clear as a bell. Thanks, Emilien!
I've revised the doc text plus I've made the following additions: In the Upgrade Guide, under "Overcloud Parameters", I've added a note about the new fixed VIP params. Draft content here: https://doc-stage.usersys.redhat.com/documentation/en-us/red_hat_openstack_platform/11/html/upgrading_red_hat_openstack_platform/chap-director-major-upgrade#sect-Updating_Overcloud_Templates In the Ext LB Guide, I've revised the environment file with the ex lb to include the new fixed VIP params: https://doc-stage.usersys.redhat.com/documentation/en-us/red_hat_openstack_platform/11/html/external_load_balancing_for_the_overcloud/configuring_the_overcloud#configuring-load-balancing-options @Marius and @Emilien, how do these additions looks? Is there anything further we need to add?
Dan: I've reviewed your documentation change and it looks good to me, thanks!
Thanks, Emilien!