Hello, Custom network VIPs are stored in a hash called network_virtual_ips, but it's impossible to access the hash's keys with hiera from hieradata. This makes it impossible to use these VIPs in haproxy.cfg. ~~~ [root@overcloud-controller-0 ~]# hiera -c /etc/puppet/hiera.yaml network_virtual_ips {"internal_api"=>{"index"=>3, "ip_address"=>"172.17.0.15"}, "storage"=>{"index"=>1, "ip_address"=>"172.20.0.16"}, "storage_mgmt"=>{"index"=>2, "ip_address"=>"172.19.0.19"}} [root@overcloud-controller-0 ~]# [root@overcloud-controller-0 ~]# hiera -c /etc/puppet/hiera.yaml network_virtual_ips.internal_api nil ~~~ I can conclude that this is indeed a problem with the templates' logic in combination with an old hiera version: When using hiera 1.3.4 (the RHEL 7 / OSP 13 default), hash indexes cannot be looked up. When using a more recent version of hiera, this works: I tested these lookups with the newest puppet lab container: ~~~ docker run --name puppet --hostname puppet puppet/puppetserver ~~~ ~~~ [root@overcloud-controller-0 ~]# docker exec -it puppet /bin/bash root@puppet:/# root@puppet:/# hiera -v 3.6.0 root@puppet:/# ~~~ And I ran: ~~~ cat<<'EOF'>/etc/puppet/hiera.yaml --- :backends: - json :json: :datadir: "/etc/puppet/hieradata" :hierarchy: - "common" EOF cat<<'EOF'>/etc/puppet/hieradata/common.json { "rh_test0": { "a": "b", "c":"d" }, "rh_test1": "%{hiera('rh_test0.a')}", "rh_test2": "%{hiera('rh_test0.c')}" } EOF ~~~ And indeed, I could look up the hash key's value: ~~~ root@puppet:/# hiera -c /etc/puppet/hiera.yaml rh_test0 {"a"=>"b", "c"=>"d"} root@puppet:/# hiera -c /etc/puppet/hiera.yaml rh_test1 b root@puppet:/# hiera -c /etc/puppet/hiera.yaml rh_test2 d root@puppet:/# ~~~ --------------------------------------------------------------------- More background: Juniper created this file: templates/openstack-tripleo-heat-templates/docker/services/contrail/contrail-config.yaml ~~~ 81 outputs: 82 role_data: 83 description: Role data for the Contrail Config service. 84 value: 85 service_name: contrail_config 86 config_settings: 87 map_merge: 88 - get_attr: [ContrailBase, role_data, config_settings] 89 - tripleo.contrail_config.firewall_rules: 90 '118 contrail_config': 91 dport: 92 - 8082 93 - 8084 94 - 8087 95 - 8088 96 - 8096 97 - 8100 98 - 8101 99 service_config_settings: 100 map_merge: 101 - {get_attr: [ContrailBase, role_data, service_config_settings]} 102 - haproxy: 103 tripleo.contrail_config.haproxy_endpoints: 104 contrail_config: 105 map_merge: 106 - public_ssl_port: 18082 107 internal_ip: 108 str_replace: 109 template: "%{hiera('NETWORK_virtual_ip')}" 110 params: 111 NETWORK: {get_param: [ServiceNetMap, ContrailConfigNetwork]} 112 service_port: 8082 113 service_network: {get_param: [ServiceNetMap, ContrailConfigNetwork]} 114 - {get_attr: [ContrailBase, role_data, haproxy_config_settings_api]} 115 contrail_env_file: &contrail_env_file 116 "/etc/contrail/common_config.env" 117 docker_config: ~~~ In the above, see: ~~~ 108 str_replace: 109 template: "%{hiera('NETWORK_virtual_ip')}" 110 params: 111 NETWORK: {get_param: [ServiceNetMap, ContrailConfigNetwork]} ~~~ NETWORK will be replaced with contrailctl, as instructed in their network_data.yaml: ~~~ #CTL-CONTRAIL - name: ContrailCtl name_lower: contrailctl vip: true ip_subnet: 172.31.205.0/27 allocation_pools: [{'start': '172.31.205.4', 'end': '172.31.205.5'}] vlan: 1455 ~~~ The problem with Juniper's approach is that they lookup ctlplane_virtual_ip. This hiera variable does not exist. I looked into our templates and the VIP will be stored in hiera variable network_virtual_ips - that's done in /usr/share/openstack-tripleo-heat-templates/puppet/all-nodes-config.j2.yaml: ~~~ 175 public_virtual_ip: {get_param: [NetVipMap, {get_param: ExternalNetName}]} 176 controller_virtual_ip: {get_param: [NetVipMap, ctlplane]} 177 {%- if 'InternalApi' in _networks %} 178 # the internal_api_virtual_ip is needed for contrail only 179 internal_api_virtual_ip: {get_param: [NetVipMap, {get_param: InternalApiNetName}]} 180 {%- endif %} 181 network_virtual_ips: 182 {%- for network in networks if network.vip|default(false) %} 183 # External virtual ip is currently being handled separately as public_virtual_ip. 184 # Likewise, optional StorageNFS virtual ip is handled separately as ganesha_vip. 185 {%- if network.name != 'External' and network.name != 'StorageNFS' %} 186 {{network.name_lower}}: 187 ip_address: {get_param: [NetVipMap, {get_param: {{network.name}}NetName}]} 188 index: {{loop.index}} 189 {%- endif %} 190 {%- endfor %} 191 redis_vip: {get_param: RedisVirtualIP} 192 {%- for network in networks if network.name == 'StorageNFS' %} 193 ganesha_vip: {get_param: [NetVipMap, {get_param: StorageNFSNetName}]} 194 {%- endfor %} ~~~ So this will be stored in here: ~~~ [root@overcloud-controller-0 ~]# hiera -c /etc/puppet/hiera.yaml network_virtual_ips {"internal_api"=>{"index"=>3, "ip_address"=>"172.17.0.15"}, "storage"=>{"index"=>1, "ip_address"=>"172.20.0.16"}, "storage_mgmt"=>{"index"=>2, "ip_address"=>"172.19.0.19"}, "contrailctl"=> (...) } [root@overcloud-controller-0 ~]# ~~~ If we go back to file templates/openstack-tripleo-heat-templates/docker/services/contrail/contrail-config.yaml: ~~~ 108 str_replace: 109 template: "%{hiera('NETWORK_virtual_ip')}" 110 params: 111 NETWORK: {get_param: [ServiceNetMap, ContrailConfigNetwork]} ~~~ One would need to do a hiera lookup for: network_virtual_ips.contrailctl as follows: ~~~ 108 str_replace: 109 template: "%{hiera('network_virtual_ips.NETWORK')}" 110 params: 111 NETWORK: {get_param: [ServiceNetMap, ContrailConfigNetwork]} ~~~ Note that the above data is simply written to /etc/puppet/hieradata, e.g.: ~~~ [root@overcloud-controller-0 ~]# grep hiera /etc/puppet/hieradata/service_configs.json | head "aodh::api::host": "%{hiera('fqdn_internal_api')}", "%{hiera('mysql_bind_host')}" "aodh::wsgi::apache::bind_host": "%{hiera('internal_api')}", "aodh::wsgi::apache::servername": "%{hiera('fqdn_internal_api')}", "apache::ip": "%{hiera('internal_api')}", "%{hiera('apache_remote_proxy_ips_network')}" "apache_remote_proxy_ips_network": "%{hiera('internal_api_subnet')}", "%{hiera('mysql_bind_host')}" "ceilometer::host": "%{hiera('fqdn_canonical')}", "cinder::api::bind_host": "%{hiera('fqdn_internal_api')}", ~~~ From there, hiera executes variable interpolation. Unfortunately, it's *impossible* to access network_virtual_ips.contrailctl within hiera with hiera-1.3.4. https://stackoverflow.com/questions/27860333/complex-hiera-lookup-not-working explains this issue. In later versions of hiera, this works: https://puppet.com/docs/hiera/3.2/variables.html#interpolating-hash-or-array-elements But not with what's currently available in OSP Director 13. A small test in my lab: ~~~ [root@overcloud-controller-0 ~]# hiera -v 1.3.4 [root@overcloud-controller-0 ~]# cat /etc/puppet/hieradata/docker_puppet.json { "rh_test0": "%{hiera('network_virtual_ips')}", "rh_test1": "%{hiera('network_virtual_ips.internal_api')}", "rh_test2": { "a": "b", "c":"d" }, "rh_test3": "%{hiera('rh_test2.a')}" } [root@overcloud-controller-0 ~]# hiera -c /etc/puppet/hiera.yaml rh_test0 {"internal_api"=>{"index"=>3, "ip_address"=>"172.17.0.15"}, "storage"=>{"index"=>1, "ip_address"=>"172.20.0.16"}, "storage_mgmt"=>{"index"=>2, "ip_address"=>"172.19.0.19"}} [root@overcloud-controller-0 ~]# hiera -c /etc/puppet/hiera.yaml rh_test1 [root@overcloud-controller-0 ~]# hiera -c /etc/puppet/hiera.yaml rh_test2 {"a"=>"b", "c"=>"d"} [root@overcloud-controller-0 ~]# hiera -c /etc/puppet/hiera.yaml rh_test3 ~~~ Possible solutions to this issue: * One solution would be to use a more recent version of hiera. * Another solution would be to instead expose all VIPs as key -> value pairs, the same as e.g. for controller_virtual_ip: /usr/share/openstack-tripleo-heat-templates/puppet/all-nodes-config.j2.yaml ~~~ (...) {%- for network in networks if network.vip|default(false) %} {%- if network.name != 'External' and network.name != 'StorageNFS' %} {{network.name_lower}}_vip: {get_param: [NetVipMap, {get_param: {{network.name}}NetName}]} {%- endif %} {%- endfor %} (...) ~~~ Thanks, - Andreas
And from the customer's environment, we get the same: root@controller01 ~]# hiera -c /etc/puppet/hiera.yaml network_virtual_ips {"contrailctl"=>{"index"=>4, "ip_address"=>"192.168.215.12"}, "internalapi"=>{"index"=>1, "ip_address"=>"192.168.212.12"}, "storagectl"=>{"index"=>3, "ip_address"=>"192.168.214.12"}} [root@controller01 ~]# hiera -c /etc/puppet/hiera.yaml network_virtual_ips.contrailctl nil
Hi Please can you provide me an update on this bug or when this is planed to be fixed? Kind Regards, Billy
I wonder if dropping hiera is the better approach. If we add the below change in tripleo, L107-L11 of file: templates/openstack-tripleo-heat-templates/docker/services/contrail/contrail-config.yaml can be replaced with: 107 internal_ip: {get_param: ServiceData, net_vip_map, {get_param: [ServiceNetMap, ContrailConfigNetwork]}} diff --git a/overcloud.j2.yaml b/overcloud.j2.yaml index cdd2cc0e9..c5ebd4194 100644 --- a/overcloud.j2.yaml +++ b/overcloud.j2.yaml @@ -544,6 +544,7 @@ resources: ServiceNetMap: {get_attr: [ServiceNetMap, service_net_map]} ServiceData: net_cidr_map: {get_attr: [NetCidrMapValue, value]} + net_vip_map: {get_attr: [VipMap, net_ip_map]} EndpointMap: {get_attr: [EndpointMapData, value]} DefaultPasswords: {get_attr: [DefaultPasswords, passwords]} RoleName: {{role.name}} Juniper created this file: templates/openstack-tripleo-heat-templates/docker/services/contrail/contrail-config.yaml ~~~ 99 service_config_settings: 100 map_merge: 101 - {get_attr: [ContrailBase, role_data, service_config_settings]} 102 - haproxy: 103 tripleo.contrail_config.haproxy_endpoints: 104 contrail_config: 105 map_merge: 106 - public_ssl_port: 18082 107 internal_ip: 108 str_replace: 109 template: "%{hiera('NETWORK_virtual_ip')}" 110 params: 111 NETWORK: {get_param: [ServiceNetMap, ContrailConfigNetwork]} 112 service_port: 8082 113 service_network: {get_param: [ServiceNetMap, ContrailConfigNetwork]} 114 - {get_attr: [ContrailBase, role_data, haproxy_config_settings_api]} ~~~ @Billy, do you have an environment where you can try this out? Would the above change fix your problem? We may consider extending the patch to also include redis and ovn_dbs vip's, i.e: diff --git a/overcloud.j2.yaml b/overcloud.j2.yaml index cdd2cc0e9..c5ebd4194 100644 --- a/overcloud.j2.yaml +++ b/overcloud.j2.yaml @@ -544,6 +544,7 @@ resources: ServiceNetMap: {get_attr: [ServiceNetMap, service_net_map]} ServiceData: net_cidr_map: {get_attr: [NetCidrMapValue, value]} + net_vip_map: + map_merge: + - {get_attr: [VipMap, net_ip_map]} + - redis: {get_attr: [RedisVirtualIP, ip_address]} + - ovn_dbs: {get_attr: [OVNDBsVirtualIP, ip_address]} EndpointMap: {get_attr: [EndpointMapData, value]} DefaultPasswords: {get_attr: [DefaultPasswords, passwords]} RoleName: {{role.name}}
The customer reported that Template validation failed. The deploy.log file contains run this run's log.
Created attachment 1654976 [details] overcloud.j2.yaml
Created attachment 1654978 [details] deploy.log
Created attachment 1654979 [details] contrail-config.yaml
contrail-config.yaml: 99 service_config_settings: 100 map_merge: 101 - {get_attr: [ContrailBase, role_data, service_config_settings]} 102 - haproxy: 103 tripleo.contrail_config.haproxy_endpoints: 104 contrail_config: 105 map_merge: 106 - public_ssl_port: 18082 107 internal_ip: {get_param: ServiceData, net_vip_map, {get_param: [ServiceNetMap, ContrailConfigNetwork]}} 108 service_port: 8082 109 service_network: {get_param: [ServiceNetMap, ContrailConfigNetwork]} 110 - {get_attr: [ContrailBase, role_data, haproxy_config_settings_api]} 111 contrail_env_file: &contrail_env_file 112 "/etc/contrail/common_config.env" 113 docker_config: 114 step_2: Error message: 'OS::TripleO::Services::SkydiveAgent', 'OS::TripleO::Services::SkydiveAnalyzer', 'OS::TripleO::Services::Snmp', 'OS::TripleO::Services::Sshd', 'OS::TripleO::Services::Tacker', 'OS::TripleO::Services::Timezone', 'OS::TripleO::Services::TripleoFirewall', 'OS::TripleO::Services::TripleoPackages', 'OS::TripleO::Services::Tuned', 'OS::TripleO::Services::Zaqar', 'OS::TripleO::Services::Ptp'], 'default_route_networks': ['MgmtCtl'], 'CountDefault': 1, 'name': 'Controller', 'tags': ['primary', 'controller'], 'networks': ['MgmtCtl', 'InternalApi', 'External', 'StorageCtl', 'ContrailCtl'], 'description': 'Controller role that has all the controler services loaded and handles\ Database, Messaging and Network functions.\ (...) 'OS::TripleO::Services::TripleoFirewall', 'OS::TripleO::Services::TripleoPackages', 'OS::TripleO::Services::Tuned', 'OS::TripleO::Services::Ptp', 'OS::TripleO::Services::ContrailCertmongerUser'], 'description': 'Virtual perf\ ', 'networks': ['MgmtCpt', 'InternalCpt', 'StorageCpt', 'CephCpt', 'ContrailCpt'], 'name': 'hciComputePerfVirtual'}] Waiting for messages on queue 'tripleo' with no timeout. Action tripleo.parameters.update execution failed: Error validating environment for plan overcloud: ERROR: Internal Error Traceback (most recent call last): File \"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py\", line 166, in _process_incoming res = self.dispatcher.dispatch(message) File \"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py\", line 220, in dispatch return self._do_dispatch(endpoint, method, ctxt, args) File \"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py\", line 190, in _do_dispatch result = func(ctxt, **new_args) File \"/usr/lib/python2.7/site-packages/osprofiler/profiler.py\", line 158, in wrapper result = f(*args, **kwargs) File \"/usr/lib/python2.7/site-packages/heat/common/context.py\", line 422, in wrapped return func(self, ctx, *args, **kwargs) File \"/usr/lib/python2.7/site-packages/heat/engine/service.py\", line 1283, in validate_template result.update(stack.get_nested_parameters(filter_parameter)) File \"/usr/lib/python2.7/site-packages/heat/engine/stack.py\", line 1094, in get_nested_parameters params.update(nested.get_nested_parameters(filter_func)) File \"/usr/lib/python2.7/site-packages/heat/engine/stack.py\", line 1094, in get_nested_parameters params.update(nested.get_nested_parameters(filter_func)) File \"/usr/lib/python2.7/site-packages/heat/engine/stack.py\", line 1077, in get_nested_parameters nested = rsrc.get_nested_parameters_stack() File \"/usr/lib/python2.7/site-packages/heat/engine/resources/stack_resource.py\", line 210, in get_nested_parameters_stack child_template = self.child_template() File \"/usr/lib/python2.7/site-packages/heat/engine/resources/template_resource.py\", line 179, in child_template self.template_url) File \"/usr/lib/python2.7/site-packages/heat/common/template_format.py\", line 116, in parse tpl = simple_parse(tmpl_str, tmpl_url) File \"/usr/lib/python2.7/site-packages/heat/common/template_format.py\", line 79, in simple_parse raise ValueError(msg) ValueError: Error parsing template http://192.168.210.10:8080/v1/AUTH_a5040294aee74ca69f85aa8689477295/overcloud/user-files/home/stack/templates/openstack-tripleo-heat-templates/docker/services/contrail/contrail-config.yaml while constructing a mapping in \"<unicode string>\", line 107, column 36: internal_ip: {get_param: ServiceData, net_vip ... ^ found unacceptable key (unhashable type: 'dict') in \"<unicode string>\", line 107, column 74: ... aram: ServiceData, net_vip_map, {get_param: [ServiceNetMap, Cont ... ^ real\t2m16.056s user\t0m8.529s sys\t0m0.663s", "stderr_lines": ["The disable_upgrade_deployment flag is not set in the roles file. This flag is expected when you have a nova-compute or swift-storage role. Please check the contents of the roles file: [{'ServicesDefault': ['OS::TripleO::Services::Aide', 'OS::TripleO::Services::AodhApi', 'OS::TripleO::Services::AodhEvaluator', 'OS::TripleO::Services::AodhListener', 'OS::TripleO::Services::AodhNotifier', 'OS::TripleO::Services::AuditD', 'OS::TripleO::Services::BarbicanApi', 'OS (...)
Created attachment 1654992 [details] contrail-analytics.yaml
Andreas, I could be mistaken, but I believe there needs to be an additional pair of nested square brackets in Harald's solution, so: If we add the below change in tripleo, L107-L11 of file: templates/openstack-tripleo-heat-templates/docker/services/contrail/contrail-config.yaml can be replaced with: 107 internal_ip: {get_param: ServiceData, net_vip_map, {get_param: [ServiceNetMap, ContrailConfigNetwork]}} Should be: 107 internal_ip: {get_param: [ServiceData, net_vip_map, {get_param: [ServiceNetMap, ContrailConfigNetwork]}]} Can you give that a try?
Seems that all's good with that last modification :-) --- #33 (Customer) Make PrivatePrivate Helps Resolution? 0 Created By: SOIYFFIDINE CHEIK OMAR (1/27/2020 11:01 AM) Hello, It works. When do you plan to integrate the following variable? On which version exactly. For inform juniper to use this variable in their template. overcloud.j2.yaml net_vip_map: {get_attr: [VipMap, net_ip_map]} templates/openstack-tripleo-heat-templates/docker/services/contrail/contrail-config.yaml internal_ip: {get_param: [ServiceData, net_vip_map, {get_param: [ServiceNetMap, ContrailConfigNetwork]}]} Regards,
Hi, For Juniper to use this variable then Redhat need to release new templates with the new variable. We can change our template only when they release new package. (It will not be possible to use our new template with older templates from RH.) Kind Regards, Billy
The patch has merged in upstream master, and I have proposed patches upstream to backport this to Queens. My intention is also to backport this to downstream Red Hat OSP. It will be available in OSP-13 and later releases. I cannot give you a date at this time, but we will update the bug once we have more information. Best Regards, Harald
Great! The customer (and I) will surely appreciate it! :-) Thanks, Andreas
Hi @Harald Jensås Please let me know once this is back-ported to Queens Kind Regards, Billy
Note that this was backported to Queens on 1/28 - https://review.opendev.org/#/c/704286/.
Verified that FIV is in 13z11 and regressions pass. The bug itself was verified by customer in Comment 13.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0760