Bug 1416355 - OSP11: environment file used for external loadbalancer deployment uses new FixedIPs parameters
Summary: OSP11: environment file used for external loadbalancer deployment uses new Fi...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: documentation
Version: 11.0 (Ocata)
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: ga
: 11.0 (Ocata)
Assignee: Dan Macpherson
QA Contact: RHOS Documentation Team
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-01-25 10:49 UTC by Marius Cornea
Modified: 2017-05-18 08:04 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Overclouds with external load balancing now use new *FixedIPs parameters. This deprecates VIP environment files. When deploying an overcloud with external load balancing deployment, ensure the external load balancing environment files do not include: OS::TripleO::Network::Ports::NetVipMap: ../network/ports/net_vip_map_external_v6.yaml OS::TripleO::Network::Ports::ExternalVipPort: ../network/ports/noop.yaml OS::TripleO::Network::Ports::InternalApiVipPort: ../network/ports/noop.yaml OS::TripleO::Network::Ports::StorageVipPort: ../network/ports/noop.yaml OS::TripleO::Network::Ports::StorageMgmtVipPort: ../network/ports/noop.yaml OS::TripleO::Network::Ports::RedisVipPort: ../network/ports/from_service_v6.yaml The new parameters (and their default values) are: - ControlFixedIPs: [{'ip_address':'192.0.2.251'}] - PublicVirtualFixedIPs: [{'ip_address':'10.0.0.251'}] - InternalApiVirtualFixedIPs: [{'ip_address':'172.16.2.251'}] - StorageVirtualFixedIPs: [{'ip_address':'172.16.1.251'}] - StorageMgmtVirtualFixedIPs: [{'ip_address':'172.16.3.251'}] - RedisVirtualFixedIPs: [{'ip_address':'172.16.2.252'}] As a result, the director now defines the VIPs with *FixedIPs parameters and configures the database and API endpoints accordingly.
Clone Of:
Environment:
Last Closed: 2017-05-18 08:04:49 UTC
Target Upstream Version:


Attachments (Terms of Use)
os-collect-config log (804.39 KB, text/x-vhdl)
2017-03-02 09:58 UTC, Marius Cornea
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1430757 0 unspecified CLOSED [Docs][Upgrades] OSP10 -> OSP11 upgrade fails for deployments using external loadbalancer 2021-02-22 00:41:40 UTC

Internal Links: 1430757

Description Marius Cornea 2017-01-25 10:49:28 UTC
Description of problem:
Overcloud deployment using external loadbalancer cluster fails with:

overcloud.AllNodesDeploySteps.ControllerDeployment_Step2.0:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: 6a0471af-8cf0-4382-be11-fbe4c33c75dc
  status: CREATE_FAILED
  status_reason: |
    Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 1
  deploy_stdout: |
    Notice: hiera(): Cannot load backend module_data: cannot load such file -- hiera/backend/module_data_backend
    Notice: Scope(Class[Tripleo::Firewall::Post]): At this stage, all network traffic is blocked.
    Notice: Compiled catalog for overcloud-controller-0.localdomain in environment production in 8.20 seconds
  deploy_stderr: |
    ...
    Warning: ModuleLoader: module 'timezone' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules
       (file & line not available)
    Warning: Unknown variable: 'osd_caps'. at /etc/puppet/modules/ceph/manifests/key.pp:101:25
    Warning: Unknown variable: 'mds_caps'. at /etc/puppet/modules/ceph/manifests/key.pp:101:36
    Warning: Unknown variable: 'mds_caps'. at /etc/puppet/modules/ceph/manifests/key.pp:101:36
    Warning: ModuleLoader: module 'openstacklib' has unresolved dependencies - it will only see those that are resolved. Use 'puppet module list --tree' to see information about modules
       (file & line not available)
    Warning: Could not find resource 'Mysql_user[nova_api@cell0]' in parameter 'require'
       (at /etc/puppet/modules/openstacklib/manifests/db/mysql/host_access.pp:59)
    Error: Could not find dependency Mysql_user[nova_api@cell0] for Mysql_grant[nova_api@cell0/nova_api_cell0.*] at /etc/puppet/modules/openstacklib/manifests/db/mysql/host_access.pp:56
    (truncated, view all with --long)

Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-6.0.0-0.20170120220933.36d493c.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy overcloud with custom roles and using an external load balancer:

source ~/stackrc
export THT=/usr/share/openstack-tripleo-heat-templates/

openstack overcloud deploy --templates $THT \
-r ~/openstack_deployment/roles/roles_data.yaml \
-e $THT/environments/puppet-pacemaker.yaml \
-e $THT/environments/network-isolation.yaml \
-e $THT/environments/network-management.yaml \
-e $THT/environments/storage-environment.yaml \
-e $THT/environments/external-loadbalancer-vip.yaml \
-e ~/openstack_deployment/environments/nodes.yaml \
-e ~/openstack_deployment/environments/network-environment.yaml \
-e ~/openstack_deployment/environments/disk-layout.yaml \
-e ~/openstack_deployment/environments/neutron-settings.yaml \
-e ~/openstack_deployment/environments/external-lb.yaml \
--log-file overcloud_deployment.log &> overcloud_install.log


Actual results:
Deployment fails.

Expected results:
Deployment passes.

Additional info:
Environment files:
http://paste.openstack.org/show/596412/

Comment 1 Emilien Macchi 2017-01-25 16:41:34 UTC
This bug should not be here anymore, we removed the nova_cell thing in puppet-nova (code was bad). It's a build problem I think, where puppet-tripleo needs an update.

Comment 2 Mike Burns 2017-01-25 16:49:01 UTC
(In reply to Emilien Macchi from comment #1)
> This bug should not be here anymore, we removed the nova_cell thing in
> puppet-nova (code was bad). It's a build problem I think, where
> puppet-tripleo needs an update.

Can you link to the patch in puppet-nova and move to POST?

Comment 3 Mike Burns 2017-01-25 16:49:14 UTC
or puppet-tripleo

Comment 4 Emilien Macchi 2017-03-01 20:09:35 UTC
Marius, can you check this one is good?

Thanks!

Comment 5 Marius Cornea 2017-03-02 09:57:44 UTC
(In reply to Emilien Macchi from comment #4)
> Marius, can you check this one is good?
> 
> Thanks!

No, the error is till there. Please note that the Nova control plane services run on a different role than controller so this could be related. 

This is the roles_data.yaml:
http://paste.openstack.org/show/601095/


Attaching the os-collect-config log on controller which shows the error.

Comment 6 Marius Cornea 2017-03-02 09:58:33 UTC
Created attachment 1259069 [details]
os-collect-config log

Comment 7 Emilien Macchi 2017-03-22 17:33:25 UTC
Looking at the environment that Marius provided, MySQLClient service is missing, so Puppet fails to create the mysql users on ServiceApi role:

Error: Could not find dependency Mysql_user[nova@cell0] for Mysql_grant[nova@cell0/nova_cell0.*]

Marking this bug as invalid for now but feel free to re-open it if there is something else.

Comment 8 Emilien Macchi 2017-03-23 19:50:02 UTC
The problem is related to your environment, you need to comment:

  OS::TripleO::Network::Ports::NetVipMap: /usr/share/openstack-tripleo-heat-templates/network/ports/net_vip_map_external.yaml
  OS::TripleO::Network::Ports::ExternalVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml
  OS::TripleO::Network::Ports::InternalApiVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml
  OS::TripleO::Network::Ports::StorageVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml
  OS::TripleO::Network::Ports::StorageMgmtVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml
  OS::TripleO::Network::Ports::RedisVipPort: /usr/share/openstack-tripleo-heat-templates/network/ports/from_service.yaml

We don't want these noop files otherwise we'll use the Control Plane VIP which is... empty.

Please try again without these lines and tell me if it helped.

Also, I'm wondering if we need a release note about that (with good documentation for our users.

Comment 9 Marius Cornea 2017-03-23 20:23:28 UTC
(In reply to Emilien Macchi from comment #8)
> The problem is related to your environment, you need to comment:
> 
>   OS::TripleO::Network::Ports::NetVipMap:
> /usr/share/openstack-tripleo-heat-templates/network/ports/
> net_vip_map_external.yaml
>   OS::TripleO::Network::Ports::ExternalVipPort:
> /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml
>   OS::TripleO::Network::Ports::InternalApiVipPort:
> /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml
>   OS::TripleO::Network::Ports::StorageVipPort:
> /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml
>   OS::TripleO::Network::Ports::StorageMgmtVipPort:
> /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml
>   OS::TripleO::Network::Ports::RedisVipPort:
> /usr/share/openstack-tripleo-heat-templates/network/ports/from_service.yaml
> 
> We don't want these noop files otherwise we'll use the Control Plane VIP
> which is... empty.
> 
> Please try again without these lines and tell me if it helped.
> 
> Also, I'm wondering if we need a release note about that (with good
> documentation for our users.

Thanks, it looks like this is a change introduced in Ocata by https://github.com/openstack/tripleo-heat-templates/commit/a3f03eb307797ac5eef1251b9252e642db326e07

I'll test it tomorrow in the morning and get back with my results. We probably need to add this to the docs.

Comment 10 Marius Cornea 2017-03-24 11:47:00 UTC
Deployment completed successfully with the following environment file. I am switching this BZ to docs so we make sure we cover the changes in the product documentation.


resource_registry:

  OS::TripleO::Controller::Ports::ExternalPort: /usr/share/openstack-tripleo-heat-templates/network/ports/external_from_pool.yaml
  OS::TripleO::Controller::Ports::InternalApiPort: /usr/share/openstack-tripleo-heat-templates/network/ports/internal_api_from_pool.yaml
  OS::TripleO::Controller::Ports::StoragePort: /usr/share/openstack-tripleo-heat-templates/network/ports/storage_from_pool.yaml
  OS::TripleO::Controller::Ports::StorageMgmtPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml
  OS::TripleO::Controller::Ports::TenantPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml
  OS::TripleO::Controller::Ports::ManagementPort: /usr/share/openstack-tripleo-heat-templates/network/ports/management_from_pool.yaml

  OS::TripleO::ServiceApi::Ports::ExternalPort: /usr/share/openstack-tripleo-heat-templates/network/ports/noop.yaml
  OS::TripleO::ServiceApi::Ports::InternalApiPort: /usr/share/openstack-tripleo-heat-templates/network/ports/internal_api_from_pool.yaml
  OS::TripleO::ServiceApi::Ports::StoragePort: /usr/share/openstack-tripleo-heat-templates/network/ports/storage_from_pool.yaml
  OS::TripleO::ServiceApi::Ports::StorageMgmtPort: /usr/share/openstack-tripleo-heat-templates/network/ports/storage_mgmt_from_pool.yaml
  OS::TripleO::ServiceApi::Ports::TenantPort: /usr/share/openstack-tripleo-heat-templates/network/ports/tenant_from_pool.yaml
  OS::TripleO::ServiceApi::Ports::ManagementPort: /usr/share/openstack-tripleo-heat-templates/network/ports/management_from_pool.yaml
  
parameter_defaults:
  ServiceNetMap:
    KeystoneAdminApiNetwork: internal_api # allows undercloud to config endpoints

  RedisPassword: 'examplepassword'
  ControlFixedIPs: [{'ip_address':'192.168.0.120'}]
  PublicVirtualFixedIPs: [{'ip_address':'172.16.18.120'}]
  InternalApiVirtualFixedIPs: [{'ip_address':'10.0.0.120'}]
  StorageVirtualFixedIPs: [{'ip_address':'10.0.0.220'}]
  StorageMgmtVirtualFixedIPs: [{'ip_address':'10.0.1.120'}]
  RedisVirtualFixedIPs: [{'ip_address':'10.0.0.119'}]
  ControllerIPs:
    external:
    - 172.16.18.121 
    - 172.16.18.122
    - 172.16.18.123
    internal_api:
    - 10.0.0.121
    - 10.0.0.122
    - 10.0.0.123
    storage:
    - 10.0.0.221
    - 10.0.0.222
    - 10.0.0.223
    storage_mgmt:
    - 10.0.1.101 
    - 10.0.1.102
    - 10.0.1.103
    tenant:
    - 10.0.1.201
    - 10.0.1.202
    - 10.0.1.203
    management:
    - 172.16.17.201
    - 172.16.17.202
    - 172.16.17.203

  ServiceApiIPs:
    internal_api:
    - 10.0.0.124
    - 10.0.0.125
    - 10.0.0.126
    storage:
    - 10.0.0.224
    - 10.0.0.225
    - 10.0.0.226
    storage_mgmt:
    - 10.0.1.104 
    - 10.0.1.105
    - 10.0.1.106
    tenant:
    - 10.0.1.204
    - 10.0.1.205
    - 10.0.1.206
    management:
    - 172.16.17.204
    - 172.16.17.205
    - 172.16.17.206

Comment 11 Emilien Macchi 2017-03-27 14:58:09 UTC
Hey Dan,

Can you maybe help on this one?
I've added some DocText to help, let me know if you need more context.

Thanks!

Comment 12 Dan Macpherson 2017-03-27 15:25:04 UTC
Clear as a bell. Thanks, Emilien!

Comment 13 Dan Macpherson 2017-05-17 03:40:54 UTC
I've revised the doc text plus I've made the following additions:

In the Upgrade Guide, under "Overcloud Parameters", I've added a note about the new fixed VIP params.

Draft content here:

https://doc-stage.usersys.redhat.com/documentation/en-us/red_hat_openstack_platform/11/html/upgrading_red_hat_openstack_platform/chap-director-major-upgrade#sect-Updating_Overcloud_Templates

In the Ext LB Guide, I've revised the environment file with the ex lb to include the new fixed VIP params:

https://doc-stage.usersys.redhat.com/documentation/en-us/red_hat_openstack_platform/11/html/external_load_balancing_for_the_overcloud/configuring_the_overcloud#configuring-load-balancing-options

@Marius and @Emilien, how do these additions looks? Is there anything further we need to add?

Comment 14 Emilien Macchi 2017-05-17 12:54:23 UTC
Dan: I've reviewed your documentation change and it looks good to me, thanks!

Comment 15 Dan Macpherson 2017-05-17 14:06:11 UTC
Thanks, Emilien!


Note You need to log in before you can comment on or make changes to this bug.