Bug 1428986 - bigswitch RHOSP 10 deployment error (compute deployment error)
Summary: bigswitch RHOSP 10 deployment error (compute deployment error)
Keywords:
Status: CLOSED UPSTREAM
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 10.0 (Newton)
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: Upstream M2
: 12.0 (Pike)
Assignee: Emilien Macchi
QA Contact: Gurenko Alex
URL:
Whiteboard:
: 1428987 (view as bug list)
Depends On:
Blocks: 1438890 1438893
TreeView+ depends on / blocked
 
Reported: 2017-03-03 18:27 UTC by bigswitch
Modified: 2018-03-06 10:53 UTC (History)
16 users (show)

Fixed In Version: openstack-tripleo-heat-templates-7.0.0-0.20170616123155.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1438890 (view as bug list)
Environment:
Last Closed: 2017-08-16 16:03:42 UTC
Target Upstream Version:


Attachments (Terms of Use)
compute sos report (9.33 MB, application/x-xz)
2017-03-07 19:31 UTC, bigswitch
no flags Details


Links
System ID Priority Status Summary Last Updated
Launchpad 1673126 None None None 2017-03-15 15:58:48 UTC
OpenStack gerrit 446029 None MERGED Re-Add bigswitch agent support 2020-05-06 09:04:11 UTC

Description bigswitch 2017-03-03 18:27:32 UTC
Description of problem:
keystone driver error while deploying RHOSP 10 

[stack@rhosp10 ~]$ ./overcloud-deploy-hostname.sh
Removing the current plan files
Uploading new plan files
Started Mistral Workflow. Execution ID: 0896e8ff-726f-4dd0-bd8d-d2f8bfb2d1a1
Plan updated
Deploying templates in the directory /tmp/tripleoclient-FsaTLH/tripleo-heat-templates
Started Mistral Workflow. Execution ID: f177ae07-14c0-482f-906a-2181a95d32fa
{u'execution': {u'id': u'f177ae07-14c0-482f-906a-2181a95d32fa',
                u'input': {u'container': u'overcloud',
                           u'queue_name': u'32bb6e03-bf76-4092-837f-1eddf5cd18c4',
                           u'timeout': 50},
                u'name': u'tripleo.deployment.v1.deploy_plan',
                u'params': {},
                u'spec': {u'input': [u'container',
                                     {u'timeout': 240},
                                     {u'queue_name': u'tripleo'}],
                          u'name': u'deploy_plan',
                          u'tasks': {u'add_validation_ssh_key': {u'name': u'add_validation_ssh_key',
                                                                 u'on-complete': u'deploy',
                                                                 u'type': u'direct',
                                                                 u'version': u'2.0',
                                                                 u'workflow': u'tripleo.validations.v1.add_validation_ssh_key_parameter container=<% $.container %>'},
                                     u'deploy': {u'action': u'tripleo.deployment.deploy timeout=<% $.timeout %> container=<% $.container %>',
                                                 u'name': u'deploy',
                                                 u'on-error': u'set_deployment_failed',
                                                 u'on-success': u'send_message',
                                                 u'type': u'direct',
                                                 u'version': u'2.0'},
                                     u'send_message': {u'action': u'zaqar.queue_post',
                                                       u'input': {u'messages': {u'body': {u'payload': {u'execution': u'<% execution() %>',
                                                                                                       u'message': u"<% $.get('message', '') %>",
                                                                                                       u'status': u"<% $.get('status', 'SUCCESS') %>"},
                                                                                          u'type': u'tripleo.deployment.v1.deploy_plan'}},
                                                                  u'queue_name': u'<% $.queue_name %>'},
                                                       u'name': u'send_message',
                                                       u'retry': u'count=5 delay=1',
                                                       u'type': u'direct',
                                                       u'version': u'2.0'},
                                     u'set_deployment_failed': {u'name': u'set_deployment_failed',
                                                                u'on-success': u'send_message',
                                                                u'publish': {u'message': u'<% task(deploy).result %>',
                                                                             u'status': u'FAILED'},
                                                                u'type': u'direct',
                                                                u'version': u'2.0'}},
                          u'version': u'2.0'}},
 u'message': u"Failed to run action [action_ex_id=eb49f388-088a-4d38-a1c4-d2c20aa169b6, action_cls='<class 'mistral.actions.action_factory.DeployStackAction'>', attributes='{}', params='{u'container': u'overcloud', u'timeout': 50}']\n ERROR: The Parameter (KeystoneNotificationDriver) was not defined in template.",

Not sure where we specify the keystone driver 

command executed for overcloud deployment as follows

#!/bin/bash

source /home/stack/stackrc

openstack overcloud deploy --templates -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /home/stack/templates/network-environment.yaml  -e /home/stack/templates/bigswitch-config-ivs.yaml -e /home/stack/scheduler_hints_env.yaml  --compute-flavor baremetal --control-flavor baremetal --ceph-storage-flavor ceph-storage --compute-scale 1 --control-scale 1 --ceph-storage-scale 0 --neutron-network-type vlan  --neutron-disable-tunneling  --neutron-network-vlan-ranges 'datacentre:2000:2050' --ntp-server 10.8.29.9 --timeout 50

Version-Release number of selected component (if applicable):

RHOSP 10
How reproducible:
New deployment with RHOSP 10

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 bigswitch 2017-03-06 17:49:11 UTC
we are in the process of testing , would appreciate if we can get any quick help.

Comment 2 bigswitch 2017-03-06 18:03:10 UTC
Found that Bigswitch customization template seems to have changed , existing working template is not recognized in RHOSP 10

Comment 3 bigswitch 2017-03-06 18:06:00 UTC
big switch template looks as below (working in RHOSP 9 and RHOSP 8)

cat templates/bigswitch-config-ivs.yaml
# Edit bigswitch-config.yaml file. Change BCF controller IP and the credentials.

resource_registry:
  OS::TripleO::ControllerExtraConfigPre: /usr/share/openstack-tripleo-heat-templates//puppet/extraconfig/pre_deploy/controller/neutron-ml2-bigswitch.yaml
  OS::TripleO::ComputeExtraConfigPre: /usr/share/openstack-tripleo-heat-templates/puppet/extraconfig/pre_deploy/compute/neutron-ml2-bigswitch.yaml

parameters:
  controllerExtraConfig:
    neutron::agents::bigswitch::lldp_enabled: true
    neutron::agents::bigswitch::agent_enabled: false
    neutron::agents::l3::enabled: false
    neutron::agents::dhcp::enable_force_metadata: true
    neutron::agents::dhcp::enable_isolated_metadata: true
    neutron::agents::dhcp::enable_metadata_network: false
    neutron::server::l3_ha: false
  NeutronMechanismDrivers: openvswitch,bsn_ml2
  NeutronServicePlugins: bsn_l3,bsn_service_plugin
  KeystoneNotificationDriver: messaging

  NovaComputeExtraConfig:
    neutron::agents::ml2::ovs::manage_vswitch: false
    neutron::agents::ml2::ovs::enabled: false
    neutron::agents::bigswitch::lldp_enabled: false
    neutron::agents::bigswitch::agent_enabled: true

parameter_defaults:
  NeutronBigswitchRestproxyServers: 10.8.29.15:8000,10.8.29.14:8000
  NeutronBigswitchRestproxyServerAuth: admin:adminadmin
  NeutronBigswitchRestproxyAutoSyncOnFailure: True
  NeutronBigswitchRestproxyConsistencyInterval: 10
  NeutronBigswitchRestproxyNeutronId: rhosp10 #<<< Name used to prefix the tenant/project name on BCF.
  NeutronBigswitchRestproxyServerSsl: True
  NeutronBigswitchRestproxySslCertDirectory: /var/lib/neutron

Comment 4 bigswitch 2017-03-07 19:28:37 UTC
*** Bug 1428987 has been marked as a duplicate of this bug. ***

Comment 5 bigswitch 2017-03-07 19:29:48 UTC
attached bug report , for some reason compute deployment is failing

[stack@rhosp10 ~]$ heat resource-show overcloud AllNodesDeploySteps
WARNING (shell) "heat resource-show" is deprecated, please use "openstack stack resource show" instead
+------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property               | Value                                                                                                                                                                                |
+------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| attributes             | {}                                                                                                                                                                                   |
| creation_time          | 2017-03-08T02:15:46Z                                                                                                                                                                 |
| description            |                                                                                                                                                                                      |
| links                  | http://192.0.2.1:8004/v1/63c510b3a6c64f9fb71f54a8908f05a4/stacks/overcloud/dee071fc-3b92-4498-b615-a2eaf6ebeebb/resources/AllNodesDeploySteps (self)                                 |
|                        | http://192.0.2.1:8004/v1/63c510b3a6c64f9fb71f54a8908f05a4/stacks/overcloud/dee071fc-3b92-4498-b615-a2eaf6ebeebb (stack)                                                              |
|                        | http://192.0.2.1:8004/v1/63c510b3a6c64f9fb71f54a8908f05a4/stacks/overcloud-AllNodesDeploySteps-l3oo44ve42qq/fb3cb91d-da23-4db2-8284-c2f838d4e02c (nested)                            |
| logical_resource_id    | AllNodesDeploySteps                                                                                                                                                                  |
| physical_resource_id   | fb3cb91d-da23-4db2-8284-c2f838d4e02c                                                                                                                                                 |
| required_by            |                                                                                                                                                                                      |
| resource_name          | AllNodesDeploySteps                                                                                                                                                                  |
| resource_status        | CREATE_FAILED                                                                                                                                                                        |
| resource_status_reason | Error: resources.AllNodesDeploySteps.resources.ComputeDeployment_Step3.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 1 |
| resource_type          | OS::TripleO::PostDeploySteps                                                                                                                                                         |
| updated_time           | 2017-03-08T02:15:46Z                                                                                                                                                                 |
+------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Comment 6 bigswitch 2017-03-07 19:31:45 UTC
Created attachment 1260929 [details]
compute sos report

Comment 7 bigswitch 2017-03-08 00:33:14 UTC
...skipping...
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: max]/Sysctl[net.nf_conntrack_max]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Tripleo::Profile::Base::Kernel/Sysctl::Value[net.nf_connt
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: [2017-03-08 00:16:34,207] (heat-config) [DEBUG] [2017-03-07 19:16:24,633] (heat-config) [DEBUG] Running FACTER_heat_outputs_path="/var/run/heat-co
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: [2017-03-08 00:16:34,203] (heat-config) [INFO] Return code 2
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: [2017-03-08 00:16:34,203] (heat-config) [INFO] Matching apachectl 'Server version: Apache/2.4.6 (Red Hat Enterprise Linux)
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: Server built:   Aug  3 2016 08:33:27'
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: Notice: Scope(Class[Tripleo::Firewall::Post]): At this stage, all network traffic is blocked.
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: Notice: Compiled catalog for overcloud-compute-nfv-0.localdomain in environment production in 1.13 seconds
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: Notice: /Stage[setup]/Firewall::Linux::Redhat/Exec[/usr/bin/systemctl daemon-reload]/returns: executed successfully
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: Notice: /Stage[setup]/Firewall::Linux::Redhat/Service[iptables]/ensure: ensure changed 'stopped' to 'running'
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: Notice: /Stage[setup]/Tripleo::Firewall::Pre/Tripleo::Firewall::Rule[002 accept all to lo interface]/Firewall[002 accept all to lo interface]/ensu
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: Notice: /Stage[setup]/Tripleo::Firewall::Pre/Tripleo::Firewall::Rule[001 accept all icmp]/Firewall[001 accept all icmp]/ensure: created
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: Notice: /Stage[setup]/Tripleo::Firewall::Pre/Tripleo::Firewall::Rule[000 accept related established rules]/Firewall[000 accept related established
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: Notice: /Stage[setup]/Tripleo::Firewall::Pre/Tripleo::Firewall::Rule[003 accept ssh]/Firewall[003 accept ssh]/ensure: created
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: Notice: /Stage[main]/Tripleo::Vip_hosts/Host[storage_mgmt]/ensure: created
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: Notice: /Stage[main]/Tripleo::Vip_hosts/Host[ctlplane]/ensure: created
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: Notice: /Stage[main]/Tripleo::Vip_hosts/Host[internal_api]/ensure: created
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: Notice: /Stage[main]/Tripleo::Profile::Base::Kernel/Kmod::Load[nf_conntrack]/File[/etc/sysconfig/modules/nf_conntrack.modules]/ensure: created
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: Notice: /Stage[main]/Ntp::Config/File[/etc/ntp.conf]/content: content changed '{md5}913c85f0fde85f83c2d6c030ecf259e9' to '{md5}3cdad9dfd3da14f73b2
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: Notice: /File[/etc/ntp.conf]/seluser: seluser changed 'unconfined_u' to 'system_u'
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: Notice: /Stage[main]/Ntp::Config/File[/etc/ntp/step-tickers]/content: content changed '{md5}e739f854b7ef57716d68c305c0513e76' to '{md5}fe469c000d7
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: Notice: /File[/etc/ntp/step-tickers]/seluser: seluser changed 'unconfined_u' to 'system_u'
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: Notice: /Stage[main]/Ntp::Service/Service[ntp]/ensure: ensure changed 'stopped' to 'running'
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: Notice: /Stage[main]/Tripleo::Vip_hosts/Host[storage]/ensure: created
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: Notice: /Stage[main]/Tripleo::Vip_hosts/Host[external]/ensure: created
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: Notice: /Stage[main]/Main/Package_manifest[/var/lib/tripleo/installed-packages/overcloud_compute1]/ensure: created
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[snmp]/Tripleo::Firewall::Rule[127 snmp]/Firewall[127 snmp]/ensure: created
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[ntp]/Tripleo::Firewall::Rule[105 ntp]/Firewall[105 ntp]/ensure: created
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: Notice: /Stage[main]/Tripleo::Firewall/Tripleo::Firewall::Service_rules[nova_libvirt]/Tripleo::Firewall::Rule[200 nova_libvirt]/Firewall[200 nova_
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: Notice: /File[/etc/sysctl.conf]/seluser: seluser changed 'unconfined_u' to 'system_u'
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: Notice: /Stage[main]/Tripleo::Profile::Base::Kernel/Sysctl::Value[net.ipv6.conf.default.autoconf]/Sysctl[net.ipv6.conf.default.autoconf]/ensure: c
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: Notice: /Stage[main]/Tripleo::Profile::Base::Kernel/Sysctl::Value[net.ipv6.conf.default.accept_ra]/Sysctl[net.ipv6.conf.default.accept_ra]/ensure:
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: Notice: /Stage[main]/Tripleo::Profile::Base::Kernel/Sysctl::Value[net.netfilter.nf_conntrack_max]/Sysctl[net.netfilter.nf_conntrack_max]/ensure: c
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: Notice: /Stage[main]/Tripleo::Profile::Base::Kernel/Sysctl::Value[net.netfilter.nf_conntrack_max]/Sysctl_runtime[net.netfilter.nf_conntrack_max]/v
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: Notice: /Stage[main]/Tripleo::Profile::Base::Kernel/Sysctl::Value[net.ipv6.conf.all.autoconf]/Sysctl[net.ipv6.conf.all.autoconf]/ensure: created
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: Notice: /Stage[main]/Tripleo::Profile::Base::Kernel/Sysctl::Value[net.ipv6.conf.default.accept_ra]/Sysctl_runtime[net.ipv6.conf.default.accept_ra]
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: Notice: /Stage[main]/Tripleo::Profile::Base::Kernel/Sysctl::Value[net.core.netdev_max_backlog]/Sysctl[net.core.netdev_max_backlog]/ensure: created
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: Notice: /Stage[main]/Tripleo::Profile::Base::Kernel/Sysctl::Value[net.core.netdev_max_backlog]/Sysctl_runtime[net.core.netdev_max_backlog]/val: va
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: Notice: /Stage[main]/Tripleo::Profile::Base::Kernel/Sysctl::Value[net.ipv4.tcp_keepalive_intvl]/Sysctl[net.ipv4.tcp_keepalive_intvl]/ensure: creat
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: Notice: /Stage[main]/Tripleo::Profile::Base::Kernel/Sysctl::Value[net.ipv6.conf.all.accept_ra]/Sysctl[net.ipv6.conf.all.accept_ra]/ensure: created
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: Notice: /Stage[main]/Tripleo::Profile::Base::Kernel/Sysctl::Value[net.ipv6.conf.all.accept_ra]/Sysctl_runtime[net.ipv6.conf.all.accept_ra]/val: va
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: Notice: /Stage[main]/Tripleo::Profile::Base::Kernel/Sysctl::Value[net.ipv4.tcp_keepalive_time]/Sysctl[net.ipv4.tcp_keepalive_time]/ensure: created
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: Notice: /Stage[main]/Tripleo::Profile::Base::Kernel/Sysctl::Value[net.ipv4.tcp_keepalive_time]/Sysctl_runtime[net.ipv4.tcp_keepalive_time]/val: va
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: Notice: /Stage[main]/Tripleo::Profile::Base::Kernel/Sysctl::Value[net.ipv4.tcp_keepalive_probes]/Sysctl[net.ipv4.tcp_keepalive_probes]/ensure: cre
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: Notice: /Stage[main]/Tripleo::Profile::Base::Kernel/Sysctl::Value[net.ipv4.tcp_keepalive_probes]/Sysctl_runtime[net.ipv4.tcp_keepalive_probes]/val
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: Notice: /Stage[main]/Tripleo::Profile::Base::Kernel/Sysctl::Value[net.nf_conntrack_max]/Sysctl[net.nf_conntrack_max]/ensure: created
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: Notice: /Stage[main]/Tripleo::Profile::Base::Kernel/Sysctl::Value[net.nf_conntrack_max]/Sysctl_runtime[net.nf_conntrack_max]/val: val changed '262
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: Notice: /Stage[main]/Tripleo::Profile::Base::Kernel/Sysctl::Value[net.ipv6.conf.default.autoconf]/Sysctl_runtime[net.ipv6.conf.default.autoconf]/v
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: Notice: /Stage[main]/Tripleo::Profile::Base::Kernel/Sysctl::Value[kernel.pid_max]/Sysctl[kernel.pid_max]/ensure: created
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: Notice: /Stage[main]/Tripleo::Profile::Base::Kernel/Sysctl::Value[kernel.pid_max]/Sysctl_runtime[kernel.pid_max]/val: val changed '196608' to '104
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: Notice: /Stage[main]/Tripleo::Profile::Base::Kernel/Sysctl::Value[net.ipv4.tcp_keepalive_intvl]/Sysctl_runtime[net.ipv4.tcp_keepalive_intvl]/val:
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: Notice: /Stage[main]/Tripleo::Profile::Base::Kernel/Sysctl::Value[net.ipv6.conf.all.autoconf]/Sysctl_runtime[net.ipv6.conf.all.autoconf]/val: val
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: Notice: /Stage[main]/Timezone/File[/etc/localtime]/ensure: defined content as '{md5}c79354b8dbee09e62bbc3fb544853283'
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: Notice: /Stage[runtime]/Tripleo::Firewall::Post/Firewall[998 log all]/ensure: created
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: Notice: /Stage[runtime]/Tripleo::Firewall::Post/Tripleo::Firewall::Rule[999 drop all]/Firewall[999 drop all]/ensure: created
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: Notice: Finished catalog run in 3.09 seconds
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: [2017-03-08 00:16:34,203] (heat-config) [INFO] exception: connect failed
Mar 08 00:16:34 overcloud-compute-nfv-0.localdomain os-collect-config[11653]: [2017-03-08 00:16:34,203] (heat-config) [INFO] Completed /var/lib/heat-config/heat-config-puppet/4f0ac11f-7512-4ad4-a061-914b20fe6395.pp
lines 2597-2653

Comment 8 bigswitch 2017-03-09 20:57:46 UTC
@Mike: could we please prioritize this issue. We are blocked on this.

Comment 9 Dan Sneddon 2017-03-09 22:26:27 UTC
I looked through the /var/log/messages on the Compute host (from the SOS log), and found the following sections with errors. Since the first sections appear to be errors occurring in the IVS driver or BigSwitch plugin, can someone provide any insight into the causes or effects of these errors?

It appears to me that network connectivity is not properly established (in the first errors), so the Undercloud API is not reachable (the last repeated errors).

============
Mar  7 13:36:38 host-192-0-2-12 bash: ifup ivs port storage3006
Mar  7 13:36:39 host-192-0-2-12 ivs: 03-07 13:36:39.249143 [ofconnectionmanager] Controller add: tcp6:fe80::5e16:c7ff:fe00:c5d5%inband:6653 (remote, v5)
Mar  7 13:36:39 host-192-0-2-12 ivs: 03-07 13:36:39.249619 [ofconnectionmanager] cxn tcp6:fe80::5e16:c7ff:fe00:c5d5%inband:6653:0(0xbbd808): connect: Cannot assign requested address
Mar  7 13:36:39 host-192-0-2-12 ivs: 03-07 13:36:39.249638 [ofconnectionmanager] cxn tcp6:fe80::5e16:c7ff:fe00:c5d5%inband:6653:0(0xbbd808): Error trying to connect when starting, closing
Mar  7 13:36:39 host-192-0-2-12 ivs: 03-07 13:36:39.249691 [ofconnectionmanager] Controller add: tcp6:fe80::5e16:c7ff:fe00:b2bf%inband:6653 (remote, v5)
Mar  7 13:36:39 host-192-0-2-12 ivs: 03-07 13:36:39.249764 [ofconnectionmanager] cxn tcp6:fe80::5e16:c7ff:fe00:b2bf%inband:6653:0(0xbd2750): connect: Cannot assign requested address
Mar  7 13:36:39 host-192-0-2-12 ivs: 03-07 13:36:39.249775 [ofconnectionmanager] cxn tcp6:fe80::5e16:c7ff:fe00:b2bf%inband:6653:0(0xbd2750): Error trying to connect when starting, closing
Mar  7 13:36:39 host-192-0-2-12 ivs: 03-07 13:36:39.251241 [ofconnectionmanager] cxn tcp6:fe80::5e16:c7ff:fe00:c5d5%inband:6653:0(0xbbd808): connect: Cannot assign requested address
Mar  7 13:36:39 host-192-0-2-12 ivs: 03-07 13:36:39.251571 [ofconnectionmanager] cxn tcp6:fe80::5e16:c7ff:fe00:c5d5%inband:6653:0(0xbbd808): Error trying to connect when starting, closing
Mar  7 13:36:39 host-192-0-2-12 ivs: 03-07 13:36:39.251633 [ofconnectionmanager] cxn tcp6:fe80::5e16:c7ff:fe00:b2bf%inband:6653:0(0xbd2750): connect: Cannot assign requested address
Mar  7 13:36:39 host-192-0-2-12 ivs: 03-07 13:36:39.251648 [ofconnectionmanager] cxn tcp6:fe80::5e16:c7ff:fe00:b2bf%inband:6653:0(0xbd2750): Error trying to connect when starting, closing
Mar  7 13:36:39 host-192-0-2-12 ivs: 03-07 13:36:39.255201 [ofconnectionmanager] cxn tcp6:fe80::5e16:c7ff:fe00:c5d5%inband:6653:0(0xbbd808): connect: Cannot assign requested address
Mar  7 13:36:39 host-192-0-2-12 ivs: 03-07 13:36:39.255228 [ofconnectionmanager] cxn tcp6:fe80::5e16:c7ff:fe00:c5d5%inband:6653:0(0xbbd808): Error trying to connect when starting, closing
Mar  7 13:36:39 host-192-0-2-12 ivs: 03-07 13:36:39.255288 [ofconnectionmanager] cxn tcp6:fe80::5e16:c7ff:fe00:b2bf%inband:6653:0(0xbd2750): connect: Cannot assign requested address
Mar  7 13:36:39 host-192-0-2-12 ivs: 03-07 13:36:39.255303 [ofconnectionmanager] cxn tcp6:fe80::5e16:c7ff:fe00:b2bf%inband:6653:0(0xbd2750): Error trying to connect when starting, closing
Mar  7 13:36:39 host-192-0-2-12 ivs: 03-07 13:36:39.263537 [ofconnectionmanager] cxn tcp6:fe80::5e16:c7ff:fe00:c5d5%inband:6653:0(0xbbd808): connect: Cannot assign requested address
Mar  7 13:36:39 host-192-0-2-12 ivs: 03-07 13:36:39.263563 [ofconnectionmanager] cxn tcp6:fe80::5e16:c7ff:fe00:c5d5%inband:6653:0(0xbbd808): Error trying to connect when starting, closing
Mar  7 13:36:39 host-192-0-2-12 ivs: 03-07 13:36:39.263619 [ofconnectionmanager] cxn tcp6:fe80::5e16:c7ff:fe00:b2bf%inband:6653:0(0xbd2750): connect: Cannot assign requested address
Mar  7 13:36:39 host-192-0-2-12 ivs: 03-07 13:36:39.263634 [ofconnectionmanager] cxn tcp6:fe80::5e16:c7ff:fe00:b2bf%inband:6653:0(0xbd2750): Error trying to connect when starting, closing
Mar  7 13:36:39 host-192-0-2-12 ivs: 03-07 13:36:39.279794 [ofconnectionmanager] cxn tcp6:fe80::5e16:c7ff:fe00:c5d5%inband:6653:0(0xbbd808): connect: Cannot assign requested address
Mar  7 13:36:39 host-192-0-2-12 ivs: 03-07 13:36:39.279837 [ofconnectionmanager] cxn tcp6:fe80::5e16:c7ff:fe00:c5d5%inband:6653:0(0xbbd808): Error trying to connect when starting, closing
Mar  7 13:36:39 host-192-0-2-12 ivs: 03-07 13:36:39.279889 [ofconnectionmanager] cxn tcp6:fe80::5e16:c7ff:fe00:b2bf%inband:6653:0(0xbd2750): connect: Cannot assign requested address
Mar  7 13:36:39 host-192-0-2-12 ivs: 03-07 13:36:39.279901 [ofconnectionmanager] cxn tcp6:fe80::5e16:c7ff:fe00:b2bf%inband:6653:0(0xbd2750): Error trying to connect when starting, closing
Mar  7 13:36:39 host-192-0-2-12 ivs: 03-07 13:36:39.312064 [ofconnectionmanager] cxn tcp6:fe80::5e16:c7ff:fe00:c5d5%inband:6653:0(0xbbd808): connect: Cannot assign requested address
Mar  7 13:36:39 host-192-0-2-12 ivs: 03-07 13:36:39.312085 [ofconnectionmanager] cxn tcp6:fe80::5e16:c7ff:fe00:c5d5%inband:6653:0(0xbbd808): Error trying to connect when starting, closing
Mar  7 13:36:39 host-192-0-2-12 ivs: 03-07 13:36:39.312137 [ofconnectionmanager] cxn tcp6:fe80::5e16:c7ff:fe00:b2bf%inband:6653:0(0xbd2750): connect: Cannot assign requested address
Mar  7 13:36:39 host-192-0-2-12 ivs: 03-07 13:36:39.312148 [ofconnectionmanager] cxn tcp6:fe80::5e16:c7ff:fe00:b2bf%inband:6653:0(0xbd2750): Error trying to connect when starting, closing
Mar  7 13:36:39 host-192-0-2-12 ivs: 03-07 13:36:39.376807 [ofconnectionmanager] cxn tcp6:fe80::5e16:c7ff:fe00:c5d5%inband:6653:0(0xbbd808): connect: Cannot assign requested address
Mar  7 13:36:39 host-192-0-2-12 ivs: 03-07 13:36:39.376850 [ofconnectionmanager] cxn tcp6:fe80::5e16:c7ff:fe00:c5d5%inband:6653:0(0xbbd808): Error trying to connect when starting, closing
Mar  7 13:36:39 host-192-0-2-12 ivs: 03-07 13:36:39.376915 [ofconnectionmanager] cxn tcp6:fe80::5e16:c7ff:fe00:b2bf%inband:6653:0(0xbd2750): connect: Cannot assign requested address
Mar  7 13:36:39 host-192-0-2-12 ivs: 03-07 13:36:39.376927 [ofconnectionmanager] cxn tcp6:fe80::5e16:c7ff:fe00:b2bf%inband:6653:0(0xbd2750): Error trying to connect when starting, closing

============

Mar  7 13:36:39 host-192-0-2-12 bash: ifup ivs port api3005
Mar  7 13:36:39 host-192-0-2-12 systemd: Started Indigo Virtual Switch.
Mar  7 13:36:39 host-192-0-2-12 os-collect-config: + RETVAL=2
Mar  7 13:36:39 host-192-0-2-12 os-collect-config: + [[ 2 == 2 ]]
Mar  7 13:36:39 host-192-0-2-12 os-collect-config: + ping_metadata_ip
Mar  7 13:36:39 host-192-0-2-12 os-collect-config: ++ get_metadata_ip
Mar  7 13:36:39 host-192-0-2-12 os-collect-config: ++ local METADATA_IP
Mar  7 13:36:39 host-192-0-2-12 os-collect-config: ++ for URL in os-collect-config.cfn.metadata_url os-collect-config.heat.auth_url os-collect-config.request.metadata_url os-collect-config.zaqar.auth_url
Mar  7 13:36:39 host-192-0-2-12 os-collect-config: +++ os-apply-config --key os-collect-config.cfn.metadata_url --key-default '' --type raw
Mar  7 13:36:39 host-192-0-2-12 os-collect-config: +++ sed -e 's|http.*://\([^:]*\).*|\1|'
Mar  7 13:36:39 host-192-0-2-12 ivs: 03-07 13:36:39.504238 [ofconnectionmanager] cxn tcp6:fe80::5e16:c7ff:fe00:c5d5%inband:6653:0(0xbbd808): connect: Cannot assign requested address
Mar  7 13:36:39 host-192-0-2-12 ivs: 03-07 13:36:39.504263 [ofconnectionmanager] cxn tcp6:fe80::5e16:c7ff:fe00:c5d5%inband:6653:0(0xbbd808): Error trying to connect when starting, closing
Mar  7 13:36:39 host-192-0-2-12 ivs: 03-07 13:36:39.504314 [ofconnectionmanager] cxn tcp6:fe80::5e16:c7ff:fe00:b2bf%inband:6653:0(0xbd2750): connect: Cannot assign requested address
Mar  7 13:36:39 host-192-0-2-12 ivs: 03-07 13:36:39.504326 [ofconnectionmanager] cxn tcp6:fe80::5e16:c7ff:fe00:b2bf%inband:6653:0(0xbd2750): Error trying to connect when starting, closing
(0xbbd808): connect: Cannot assign requested address
Mar  7 13:36:39 host-192-0-2-12 ivs: 03-07 13:36:39.760893 [ofconnectionmanager] cxn tcp6:fe80::5e16:c7ff:fe00:c5d5%inband:6653:0(0xbbd808): Error trying to connect when starting, closing
Mar  7 13:36:39 host-192-0-2-12 ivs: 03-07 13:36:39.760961 [ofconnectionmanager] cxn tcp6:fe80::5e16:c7ff:fe00:b2bf%inband:6653:0(0xbd2750): connect: Cannot assign requested address
Mar  7 13:36:39 host-192-0-2-12 ivs: 03-07 13:36:39.760976 [ofconnectionmanager] cxn tcp6:fe80::5e16:c7ff:fe00:b2bf%inband:6653:0(0xbd2750): Error trying to connect when starting, closing
============

Mar  7 13:47:03 host-192-0-2-12 os-collect-config: [2017-03-07 18:47:03,306] (heat-config) [INFO] {"deploy_stdout": "Matching apachectl 'Server version: Apache/2.4.6 (Red Hat Enterprise Linux)\nServer built:   Aug  3 2016 08:33:27'\n", "deploy_stderr": "exception: connect failed\n\u001b[1;31mError: Must pass servers to Class[Neutron::Plugins::Ml2::Bigswitch::Restproxy] at /etc/puppet/modules/tripleo/manifests/profile/base/neutron/plugins/ml2.pp:64 on node overcloud-compute-nfv-0.localdomain\u001b[0m\n\u001b[1;31mError: Must pass servers to Class[Neutron::Plugins::Ml2::Bigswitch::Restproxy] at /etc/puppet/modules/tripleo/manifests/profile/base/neutron/plugins/ml2.pp:64 on node overcloud-compute-nfv-0.localdomain\u001b[0m\n", "deploy_status_code": 1}
Mar  7 13:47:03 host-192-0-2-12 os-collect-config: [2017-03-07 18:47:03,308] (heat-config) [DEBUG] [2017-03-07 18:46:58,747] (heat-config) [DEBUG] Running FACTER_heat_outputs_path="/var/run/heat-config/heat-config-puppet/3a5c2f45-0731-4057-89cf-5e52598ce722"  FACTER_fqdn="overcloud-compute-nfv-0.localdomain"  FACTER_deploy_config_name="ComputeDeployment_Step3"  puppet apply --detailed-exitcodes --logdest console --modulepath /etc/puppet/modules:/opt/stack/puppet-modules:/usr/share/openstack-puppet/modules /var/lib/heat-config/heat-config-puppet/3a5c2f45-0731-4057-89cf-5e52598ce722.pp
Mar  7 13:47:03 host-192-0-2-12 os-collect-config: [2017-03-07 18:47:03,302] (heat-config) [INFO] Return code 1
Mar  7 13:47:03 host-192-0-2-12 os-collect-config: [2017-03-07 18:47:03,302] (heat-config) [INFO] Matching apachectl 'Server version: Apache/2.4.6 (Red Hat Enterprise Linux)
Mar  7 13:47:03 host-192-0-2-12 os-collect-config: Server built:   Aug  3 2016 08:33:27'
Mar  7 13:47:03 host-192-0-2-12 os-collect-config: [2017-03-07 18:47:03,302] (heat-config) [INFO] exception: connect failed
Mar  7 13:47:03 host-192-0-2-12 os-collect-config: #033[1;31mError: Must pass servers to Class[Neutron::Plugins::Ml2::Bigswitch::Restproxy] at /etc/puppet/modules/tripleo/manifests/profile/base/neutron/plugins/ml2.pp:64 on node overcloud-compute-nfv-0.localdomain#033[0m
Mar  7 13:47:03 host-192-0-2-12 os-collect-config: #033[1;31mError: Must pass servers to Class[Neutron::Plugins::Ml2::Bigswitch::Restproxy] at /etc/puppet/modules/tripleo/manifests/profile/base/neutron/plugins/ml2.pp:64 on node overcloud-compute-nfv-0.localdomain#033[0m
Mar  7 13:47:03 host-192-0-2-12 os-collect-config: [2017-03-07 18:47:03,303] (heat-config) [ERROR] Error running /var/lib/heat-config/heat-config-puppet/3a5c2f45-0731-4057-89cf-5e52598ce722.pp. [1]

============

Mar  7 13:50:41 host-192-0-2-12 os-collect-config: 404 Client Error: Not Found for url: http://169.254.169.254/latest/meta-data/
Mar  7 13:50:41 host-192-0-2-12 os-collect-config: Source [ec2] Unavailable.
Mar  7 13:50:41 host-192-0-2-12 os-collect-config: 401 Client Error: Unauthorized for url: http://192.0.2.1:8080/v1/AUTH_63c510b3a6c64f9fb71f54a8908f05a4/ov-yk35isrwvo-0-6dq2js34zbb7-NovaCompute-pc2gbsan2uwv/7689d169-056d-42f0-a369-6764c936349f?temp_url_sig=598c4ba335b1ad2270365db73905b20fab48ad7c&temp_url_expires=2147483586
Mar  7 13:50:41 host-192-0-2-12 os-collect-config: Source [request] Unavailable.

[Similar messages about Client Error: Unauthorized appear several times later in the log]

============

Comment 10 bigswitch 2017-03-09 22:40:18 UTC
The errors came initially trying to get the uplink ports and trying to connect BCF controller IPV6 address, Once the Bigswitch IVS added uplink ports , then we see communication network reachability is present. we could ping to controller etc from compute node.

How ever below rest proxy should only get applied to open stack controller , should never get applied to compute , These information should be derived from big switch.yaml file which has given in the overcloud-deploy command.

Comment 11 bigswitch 2017-03-10 02:15:15 UTC
@Dan:
To reiterate, the RestProxy configs need to be applied only on the Controller nodes, but it appears like these are being referred in the Compute nodes as well (a change in behaviour from Mitaka/Liberty).

parameters:
  controllerExtraConfig:
    neutron::agents::bigswitch::lldp_enabled: true
    neutron::agents::bigswitch::agent_enabled: false
    neutron::agents::l3::enabled: false
    neutron::agents::dhcp::enable_force_metadata: true
    neutron::agents::dhcp::enable_isolated_metadata: true
    neutron::agents::dhcp::enable_metadata_network: false
    neutron::server::l3_ha: false
  NeutronMechanismDrivers: openvswitch,bsn_ml2       <------
  NeutronServicePlugins: bsn_l3,bsn_service_plugin
  KeystoneNotificationDriver: messaging

So, appears NeutronMechanismDrivers is being exposed to compute nodes as well, resulting in the ERR:
Mar  7 13:47:03 host-192-0-2-12 os-collect-config: #033[1;31mError: Must pass servers to Class[Neutron::Plugins::Ml2::Bigswitch::Restproxy] at /etc/puppet/modules/tripleo/manifests/profile/base/neutron/plugins/ml2.pp:64 on node overcloud-compute-nfv-0.localdomain#033[0m


We modified our compute.yaml template (/puppet/extraconfig/pre_deploy/compute/neutron-ml2-bigswitch.yaml) to also include the RestProxy variables in it, and we observed that the deployment proceeded further but failed on Controller Step5. No services appear to be running on the Controller but we don't observe any Error messages (other than MySQL not running) in the logs.

[root@overcloud-controller-nfv-0 heat-admin]# pcs resource show 
 ip-172.18.0.10	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-nfv-0
 ip-172.19.0.11	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-nfv-0
 ip-172.17.0.12	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-nfv-0
 Clone Set: haproxy-clone [haproxy]
     Started: [ overcloud-controller-nfv-0 ]
 Master/Slave Set: galera-master [galera]
     Masters: [ overcloud-controller-nfv-0 ]
 ip-172.17.0.16	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-nfv-0
 ip-10.8.65.201	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-nfv-0
 Clone Set: rabbitmq-clone [rabbitmq]
     Started: [ overcloud-controller-nfv-0 ]
 ip-192.0.2.13	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-nfv-0
 Master/Slave Set: redis-master [redis]
     Masters: [ overcloud-controller-nfv-0 ]
 openstack-cinder-volume	(systemd:openstack-cinder-volume):	Started overcloud-controller-nfv-0


So, it appears like the above parameters: in the template is being applied commonly to both Compute and Controller nodes, which maybe causing these issues. 

In Newton, how can we expose only certain Parameters to Nodes based on their role?

Comment 15 Dan Sneddon 2017-03-10 18:34:29 UTC
Comments inline...

(In reply to bigswitch from comment #11)
> @Dan:
> To reiterate, the RestProxy configs need to be applied only on the
> Controller nodes, but it appears like these are being referred in the
> Compute nodes as well (a change in behaviour from Mitaka/Liberty).
> 
> parameters:
>   controllerExtraConfig:
>     neutron::agents::bigswitch::lldp_enabled: true
>     neutron::agents::bigswitch::agent_enabled: false
>     neutron::agents::l3::enabled: false
>     neutron::agents::dhcp::enable_force_metadata: true
>     neutron::agents::dhcp::enable_isolated_metadata: true
>     neutron::agents::dhcp::enable_metadata_network: false
>     neutron::server::l3_ha: false
>   NeutronMechanismDrivers: openvswitch,bsn_ml2       <------
>   NeutronServicePlugins: bsn_l3,bsn_service_plugin
>   KeystoneNotificationDriver: messaging
> 
> So, appears NeutronMechanismDrivers is being exposed to compute nodes as
> well, resulting in the ERR:
> Mar  7 13:47:03 host-192-0-2-12 os-collect-config: #033[1;31mError: Must
> pass servers to Class[Neutron::Plugins::Ml2::Bigswitch::Restproxy] at
> /etc/puppet/modules/tripleo/manifests/profile/base/neutron/plugins/ml2.pp:64
> on node overcloud-compute-nfv-0.localdomain#033[0m

The NeutronMechanismDrivers parameter here is being applied to all roles. Did you mean to put that inside of the ControllerExtraConfig stanza so it would apply only to the Controller? Right now, that setting applies globally because it is included at the base level in this file.

> We modified our compute.yaml template
> (/puppet/extraconfig/pre_deploy/compute/neutron-ml2-bigswitch.yaml) to also
> include the RestProxy variables in it, and we observed that the deployment
> proceeded further but failed on Controller Step5. No services appear to be
> running on the Controller but we don't observe any Error messages (other
> than MySQL not running) in the logs.
> 
> [root@overcloud-controller-nfv-0 heat-admin]# pcs resource show 
>  ip-172.18.0.10	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-nfv-0
>  ip-172.19.0.11	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-nfv-0
>  ip-172.17.0.12	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-nfv-0
>  Clone Set: haproxy-clone [haproxy]
>      Started: [ overcloud-controller-nfv-0 ]
>  Master/Slave Set: galera-master [galera]
>      Masters: [ overcloud-controller-nfv-0 ]
>  ip-172.17.0.16	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-nfv-0
>  ip-10.8.65.201	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-nfv-0
>  Clone Set: rabbitmq-clone [rabbitmq]
>      Started: [ overcloud-controller-nfv-0 ]
>  ip-192.0.2.13	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-nfv-0
>  Master/Slave Set: redis-master [redis]
>      Masters: [ overcloud-controller-nfv-0 ]
>  openstack-cinder-volume	(systemd:openstack-cinder-volume):	Started
> overcloud-controller-nfv-0
> 
> 
> So, it appears like the above parameters: in the template is being applied
> commonly to both Compute and Controller nodes, which maybe causing these
> issues. 
> 
> In Newton, how can we expose only certain Parameters to Nodes based on their
> role?

Yes, the above parameters are being applied to all roles. You need to apply the Controller-only settings in ControllerExtraConfig.

Comment 16 Alex Schultz 2017-03-10 18:48:38 UTC
As Dan mentioned, it appears that the bigswitch-config.ivs.yaml has some formatting issues.

1) It should read 'ControllerExtraConfig' not 'controllerExtraConfig'
2) Please indent NeutronMechanismDrivers, NeutronServicePlugins and KeystoneNotificationDriver two more spaces so they are aligned under the 'ControllerExtraConfig'
3) Additionally If the NeutronBigswitchRestproxy* items should not be applied to the computes, those will also need to be moved under the 'ControllerExtraConfig' and indented properly.

My understanding is that when you're applying this yaml via -e during the deployment, the items are applied globally. In order to ensure they are only applied to a specific role, the settings need to be properly captured under the *ExtraConfig: sections. Due to the above formatting problems, those settings were being applied to everything which is resulting in some of the issues you are setting.

Please let us know if this does not resolve the issue.

Comment 17 bigswitch 2017-03-10 18:56:51 UTC
Thank you Dan and Alex for your comments. We shall try reformatting the yaml file and let you know the results.

Has the yaml parsing changed in Newton? Because in Liberty and Mitaka we have been using the same yaml as above and the parameters for NeutronMechanismDrivers/etc were being applied only to Controller nodes, not compute nodes. Is this a behavior change in Newton?

Comment 18 bigswitch 2017-03-10 19:41:52 UTC
Hi 

we modified our yaml files , But we hit the following deployment error

2017-03-11 03:36:14Z [overcloud.ObjectStorage]: CREATE_COMPLETE  state changed
2017-03-11 03:36:15Z [overcloud.Controller.0]: CREATE_FAILED  resources[0]: Property error: resources.ControllerExtraConfigPre.properties: Property NeutronBigswitchRestproxyServerAuth not assigned
2017-03-11 03:36:15Z [overcloud.Controller]: CREATE_FAILED  Resource CREATE failed: resources[0]: Property error: resources.ControllerExtraConfigPre.properties: Property NeutronBigswitchRestproxyServerAuth not assigned
2017-03-11 03:36:15Z [overcloud.CephStorage]: CREATE_COMPLETE  state changed
2017-03-11 03:36:15Z [overcloud.Compute.0]: CREATE_IN_PROGRESS  Stack CREATE started
2017-03-11 03:36:15Z [overcloud.Compute.0.NodeAdminUserData]: CREATE_IN_PROGRESS  state changed
2017-03-11 03:36:15Z [overcloud.ObjectStorageIpListMap]: CREATE_IN_PROGRESS  state changed
2017-03-11 03:36:15Z [overcloud.Compute.0.NodeUserData]: CREATE_IN_PROGRESS  state changed
2017-03-11 03:36:15Z [overcloud.ObjectStorageIpListMap]: CREATE_IN_PROGRESS  Stack CREATE started
2017-03-11 03:36:15Z [overcloud.Compute.0.UpdateConfig]: CREATE_IN_PROGRESS  state changed


Modified yaml file

resource_registry:
  OS::TripleO::ControllerExtraConfigPre: /usr/share/openstack-tripleo-heat-templates//puppet/extraconfig/pre_deploy/controller/neutron-ml2-bigswitch.yaml
  OS::TripleO::ComputeExtraConfigPre: /usr/share/openstack-tripleo-heat-templates/puppet/extraconfig/pre_deploy/compute/neutron-ml2-bigswitch.yaml

parameter_defaults:
  ControllerExtraConfig:
    neutron::agents::bigswitch::lldp_enabled: true
    neutron::agents::bigswitch::agent_enabled: false
    neutron::agents::l3::enabled: false
    neutron::agents::dhcp::enable_force_metadata: true
    neutron::agents::dhcp::enable_isolated_metadata: true
    neutron::agents::dhcp::enable_metadata_network: false
    neutron::server::l3_ha: false
    NeutronMechanismDrivers: openvswitch,bsn_ml2
    NeutronServicePlugins: bsn_l3,bsn_service_plugin
    KeystoneNotificationDriver: messaging
    NeutronBigswitchRestproxyServers: 10.8.29.15:8000,10.8.29.14:8000
    NeutronBigswitchRestproxyServerAuth: admin:adminadmin
    NeutronBigswitchRestproxyAutoSyncOnFailure: True
    NeutronBigswitchRestproxyConsistencyInterval: 10
    NeutronBigswitchRestproxyNeutronId: rhosp10 #<<< Name used to prefix the tenant/project name on BCF.
    NeutronBigswitchRestproxyServerSsl: True
    NeutronBigswitchRestproxySslCertDirectory: /var/lib/neutron

  NovaComputeExtraConfig:
    neutron::agents::ml2::ovs::manage_vswitch: false
    neutron::agents::ml2::ovs::enabled: false
    neutron::agents::bigswitch::lldp_enabled: false
    neutron::agents::bigswitch::agent_enabled: true

Comment 19 Alex Schultz 2017-03-10 20:44:07 UTC
The last error indicates that the NeutronBigswitchRestproxy* items needs to go back to where it was.  I think between 9 and 10 there was some refactoring around the ml2 profile items which is why this is a change now.  I'm not completely sure so I'm still reviewing. In the mean time please try  moving the NeutronBigswitchRestproxy* items back and leave the NeutronMechanismDrivers where they are under ControlerExtraConfig

Comment 20 Alex Schultz 2017-03-10 20:50:24 UTC
So in 9, the compute role just included ::neutron::plugins::ml2 via https://github.com/openstack/tripleo-heat-templates/blob/stable/mitaka/puppet/manifests/overcloud_compute.pp#L181

In 10, the compute role uses the tripleo::profile::base::neutron::plugins::ml2 via https://github.com/openstack/puppet-tripleo/blob/stable/newton/manifests/profile/base/neutron/plugins/ml2.pp which is pulled in via https://github.com/openstack/tripleo-heat-templates/blob/stable/newton/puppet/services/neutron-plugin-ml2.yaml

Now if you define the NeutronMechanismDrivers at a global scope, anywhere that profile is loaded will get the bigswitch configuration.  So the key for 10 is to only configure that with what you need for the specific role. So in this case for the controller NeutronMechanismDrivers should be 'openvswitch,bsn_ml2' and on the compute you'd want it to be 'openvswitch'.  This would be handled by including the different settings for each of the ExtraConfigs.  So you may still need to define it for NovaComputeExtraConfig, but with a different value.

Comment 21 bigswitch 2017-03-10 23:29:42 UTC
Thanks , we did deploy using the method mentioned above , so new yaml looks like below

parameter_defaults:
  ControllerExtraConfig:
    neutron::agents::bigswitch::lldp_enabled: true
    neutron::agents::bigswitch::agent_enabled: false
    neutron::agents::l3::enabled: false
    neutron::agents::dhcp::enable_force_metadata: true
    neutron::agents::dhcp::enable_isolated_metadata: true
    neutron::agents::dhcp::enable_metadata_network: false
    neutron::server::l3_ha: false
    NeutronMechanismDrivers: openvswitch,bsn_ml2
    NeutronServicePlugins: bsn_l3,bsn_service_plugin
    KeystoneNotificationDriver: messaging

  NovaComputeExtraConfig:
    neutron::agents::ml2::ovs::manage_vswitch: false
    neutron::agents::ml2::ovs::enabled: false
    neutron::agents::bigswitch::lldp_enabled: false
    neutron::agents::bigswitch::agent_enabled: true
    NeutronMechanismDrivers: openvswitch

#parameter_defaults:
  NeutronBigswitchRestproxyServers: 10.8.29.15:8000,10.8.29.14:8000
  NeutronBigswitchRestproxyServerAuth: admin:adminadmin
  NeutronBigswitchRestproxyAutoSyncOnFailure: True
  NeutronBigswitchRestproxyConsistencyInterval: 10
  NeutronBigswitchRestproxyNeutronId: rhosp10 #<<< Name used to prefix the tenant/project name on BCF.
  NeutronBigswitchRestproxyServerSsl: True
  NeutronBigswitchRestproxySslCertDirectory: /var/lib/neutron


With this , deployment went fine , how ever none of the big switch setting were applied to both controller and compute, so deployment completed with Openvswitch.

Comment 22 Alex Schultz 2017-03-10 23:51:50 UTC
Ok so that takes us back to the error from comment #11

Mar  7 13:47:03 host-192-0-2-12 os-collect-config: #033[1;31mError: Must pass servers to Class[Neutron::Plugins::Ml2::Bigswitch::Restproxy] at /etc/puppet/modules/tripleo/manifests/profile/base/neutron/plugins/ml2.pp:64 on node overcloud-compute-nfv-0.localdomain#033[0m

Which is failing because puppet/extraconfig/pre_deploy/compute/neutron-ml2-bigswitch.yaml doesn't have any of the Resetproxy settings as they only live in puppet/extraconfig/pre_deploy/controller/neutron-ml2-bigswitch.yaml


Specifically you still have to provide https://github.com/openstack/tripleo-heat-templates/blob/stable/newton/puppet/extraconfig/pre_deploy/controller/neutron-ml2-bigswitch.yaml#L38-L53 for the computes because the upstream neutron::plugins::ml2::bigswitch::restproxy class needs that information.

Comment 23 Alex Schultz 2017-03-11 00:01:21 UTC
Also when you say none of the bigswitch settings were applied to both controller and compute, was the bigswitch items at least applied on the controller?

Comment 24 bigswitch 2017-03-11 03:43:02 UTC
Not in the earlier deployment , Even controller did not get applied.

I made changes as per your suggestion (rest proxy setting added /pre_deploy/compute/neutron-ml2-bigswitch.yaml) 

With this change deployment went fine , and ban_ml2 and neutron config gets applied to both controller and compute ,

It seems like all the config gets applied irrpesctive of the role.

and setting in the novacompute extra config not getting applied , due to this , BSN IVS agent is not running.

Comment 25 bigswitch 2017-03-11 19:34:26 UTC
I also see that some of the setting e.g 

neutron::agents::ml2::ovs::enabled: false is avaiable in /service/packemaker/ 

and the pacemaker runs only on controller role..

Comment 26 bigswitch 2017-03-13 23:32:44 UTC
Please provide guidance/pointers as to how we may make progress from here. Thanks.

Comment 27 Alex Schultz 2017-03-14 17:26:39 UTC
Sorry for the delay, Since I'm not familiar with the expected configuration, it sounds as though in #24 it's properly configured but the IVS agent is not running? Is that correct? I'm looking in puppet-tripleo and openstack-tripleo-heat-templates and I do not see an agent profile for bigswitch. So without something to pull in the neutron::agents::bigswitch class, no agent will ever be configured.

    neutron::agents::bigswitch::lldp_enabled: false
    neutron::agents::bigswitch::agent_enabled: true 

Those two lines just set hiera data which is ignored if the class is never included. So you'll need to construct a THT file to include neutron::agents::bigswitch and make sure to apply it to the compute role.

Comment 28 bigswitch 2017-03-14 17:56:02 UTC
Alex, yes your statement is correct. But the definitions for:

    neutron::agents::bigswitch::lldp_enabled: false
    neutron::agents::bigswitch::agent_enabled: true 

can be found in 
  https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/extraconfig/pre_deploy/compute/neutron-ml2-bigswitch.yaml
  https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/compute-role.yaml

And in Newton, we are seeing that these configurations are not being picked-up from the yaml files (works fine for previous versions)


The expected behavior is as follows:

We have bsn-agent and lldp-service installed on the controller and compute nodes.

parameters:
  controllerExtraConfig:
    neutron::agents::bigswitch::lldp_enabled: true
    neutron::agents::bigswitch::agent_enabled: false

# On the Controller nodes:
#  We want the LLDP service to be running
#  We don't want bsn-agent to be running


  NovaComputeExtraConfig:
    neutron::agents::ml2::ovs::manage_vswitch: false
    neutron::agents::ml2::ovs::enabled: false
    neutron::agents::bigswitch::lldp_enabled: false
    neutron::agents::bigswitch::agent_enabled: true
# On the Compute nodes
# We don't want ovs to manage vswitch and ovs should be disabled
# We don't want lldp service to be running
# We want bsn-agent to be running

Comment 29 Alex Schultz 2017-03-14 20:10:09 UTC
So here's the sticking point, just because those lines are defined doesn't mean they actually get configured. That's just configuring the hiera data. Without something that does an 'include ::neutron::agents::bigswitch', those will never be configured on the nodes themselves.

For example if we look at the midonet implementation, there is a tripleo::profile::base::neutron::agent::midonet defined in
https://github.com/openstack/puppet-tripleo/blob/stable/newton/manifests/profile/base/neutron/agents/midonet.pp

Which gets pulled in via THT in https://github.com/openstack/tripleo-heat-templates/blob/6b8c34b27a324120f6945f7873f479d37ba2e516/puppet/services/neutron-compute-plugin-midonet.yaml and gets assigned to the compute role in https://github.com/openstack/tripleo-heat-templates/blob/6b8c34b27a324120f6945f7873f479d37ba2e516/environments/neutron-midonet.yaml#L10

I think what's missing is this piece for the bigswitch agents.  Does that make sense?

Comment 31 Alex Schultz 2017-03-14 21:21:22 UTC
No, what's missing is the inclusion of https://github.com/openstack/puppet-neutron/blob/master/manifests/agents/bigswitch.pp anywhere in the setup.  As I previously indicated:

parameters:
  controllerExtraConfig:
    neutron::agents::bigswitch::lldp_enabled: true
    neutron::agents::bigswitch::agent_enabled: false

does not actually perform a "include netron::agents::bigswitch", that only generates the hiera data so that when neutron::agents::bigswitch is include, it will properly enable/disable the agent and lldp.  As a hack to see if this is the case, you could try including "include ::neutron::agents::bigswitch" in your version of https://github.com/openstack/puppet-neutron/blob/e48383a14f943b4ccc2bec6486bc476d8d531b49/manifests/plugins/ml2/bigswitch/restproxy.pp and see if that results in the agents actually working. http://hardysteven.blogspot.com/2016/08/tripleo-deploy-artifacts-and-puppet.html has some documentation on how to upload a puppet module for testing. It  may or may not work with OSP as I've seen a big about it (I think there are some issues with 11, but I think 10 is still ok).  You could try pulling down the newton version of the puppet-neutron module and uploading it after adding the 'include ::neutron::agents::bigswitch' to neutron::plugins::ml2::bigswitch::restproxy.  


If that does in fact fix it, then we would need to properly capture this setup with a service and role data and propose changes for newton.  From an upstream perspective, I don't think the inclusion of ::neutron::agents::bigswitch in the restproxy is the correct setup, but it's a small targeted change for to see if that is ultimately the problem.

Comment 32 Alex Schultz 2017-03-14 22:43:59 UTC
Also for reference you could do something like described here https://blogs.rdoproject.org/7951/using-tripleo-composable-roles-in-openstack-newton-to-perform-extra-overcloud-configuration


Create a new service for the agent

puppet/service/bigswitch-agent.yaml
--------------------

heat_template_version: 2016-04-08
 
description: >
  Installs bigswitch agent
 
parameters:
  ServiceNetMap:
    default: {}
    description: Mapping of service_name -> network name. Typically set
                 via parameter_defaults in the resource registry.  This
                 mapping overrides those in ServiceNetMapDefaults.
    type: json
  DefaultPasswords:
    default: {}
    type: json
  EndpointMap:
    default: {}
    description: Mapping of service endpoint -> protocol. Typically set
                 via parameter_defaults in the resource registry.
    type: json

 
outputs:
  role_data:
    description: bigswitch agent
    value:
      service_name:
      step_config: |
        if hiera('step') >= 4 {
          include ::neutron::agents::bigswitch
        }



bigswitch-agent-environment.yaml
--------------------------------

resource_registery:
  OS::TripleO::Services::BigswitchAgent: puppet/service/bigswitch-agent.yaml



copy the default role_data.yaml and update to include the new services
----------------------------------------------------------------------

- name: Controller
  ...
  ServicesDefault:
    ...
    - OS::TripleO::Services::BigswitchAgent

- name: Compute
  ...
  ServicesDefault:
    ...
    - OS::TripleO::Services::BigswitchAgent


openstack overcloud deploy --templates -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /home/stack/templates/network-environment.yaml  -e /home/stack/templates/bigswitch-config-ivs.yaml -e /home/stack/scheduler_hints_env.yaml  --compute-flavor baremetal --control-flavor baremetal --ceph-storage-flavor ceph-storage --compute-scale 1 --control-scale 1 --ceph-storage-scale 0 --neutron-network-type vlan  --neutron-disable-tunneling  --neutron-network-vlan-ranges 'datacentre:2000:2050' --ntp-server 10.8.29.9 --timeout 50 -e bigswitch-agent-environment.yaml -r role_data.yaml

Comment 33 bigswitch 2017-03-15 00:16:56 UTC
Thanks Much Alex , really appreciated . Above recommendation works good (Comment 32), Now the big switch agent is running. we successfully deployed.

Here is the modified yaml file.

resource_registry:
  OS::TripleO::ControllerExtraConfigPre: /usr/share/openstack-tripleo-heat-templates//puppet/extraconfig/pre_deploy/controller/neutron-ml2-bigswitch.yaml
  OS::TripleO::ComputeExtraConfigPre: /usr/share/openstack-tripleo-heat-templates/puppet/extraconfig/pre_deploy/compute/neutron-ml2-bigswitch.yaml
  OS::TripleO::Services::BigswitchAgent: /usr/share/openstack-tripleo-heat-templates/puppet/services/bigswitch-agent.yaml
  OS::TripleO::Services::ComputeNeutronOvsAgent: OS::Heat::None
  OS::TripleO::Services::NeutronL3Agent: OS::Heat::None

parameter_defaults:
  ControllerExtraConfig:
    neutron::agents::bigswitch::lldp_enabled: true
    neutron::agents::bigswitch::agent_enabled: false
    neutron::agents::l3::enabled: false
    neutron::agents::dhcp::enable_force_metadata: true
    neutron::agents::dhcp::enable_isolated_metadata: true
    neutron::agents::dhcp::enable_metadata_network: false
    neutron::server::l3_ha: false
  NeutronMechanismDrivers: openvswitch,bsn_ml2
  NeutronServicePlugins: bsn_l3,bsn_service_plugin
  KeystoneNotificationDriver: messaging

  NovaComputeExtraConfig:
    neutron::agents::bigswitch::lldp_enabled: false
    neutron::agents::bigswitch::agent_enabled: true

Comment 34 Alex Schultz 2017-03-15 15:57:31 UTC
Excellent. I've proposed the bigswitch-agent.yaml upstream for 12. We can work on downstream backports for 10/11 as well.

Comment 35 bigswitch 2017-03-15 18:04:31 UTC
Thank you so much Alex!

There are a couple points I would like to point out here:-
1. We are still having issues with applying the RestProxy parameters to ControllerNodes only.

parameter_defaults:
  NeutronBigswitchRestproxyServers: 10.8.29.15:8000,10.8.29.14:8000
  NeutronBigswitchRestproxyServerAuth: admin:adminadmin
  NeutronBigswitchRestproxyAutoSyncOnFailure: True
  NeutronBigswitchRestproxyConsistencyInterval: 10
  NeutronBigswitchRestproxyNeutronId: rhosp10 #<<< Name used to prefix the tenant/project name on BCF.
  NeutronBigswitchRestproxyServerSsl: True
  NeutronBigswitchRestproxySslCertDirectory: /var/lib/neutron

If we leave it in global scope, we see it is applied to both Compute and Controller. If we tab indent it inside ControllerExtraConfig; the config is not being applied to Controller nodes (not seen on Compute)

2. Like the bigswitch-agent running on the compute nodes, we have a few other services that are expected to run on the computes:
Already present
 - neutron::agents::bigswitch::lldp_enabled
 - neutron::agents::bigswitch::agent_enabled
New agents:
 - neutron::agents::bigswitch::ivs_enabled
 - neutron::agents::bigswitch::nfvswitch_enabled
So, would be great if you could help guide us with how to integrate the above 2 agents as well.

Comment 36 Alex Schultz 2017-03-15 18:27:19 UTC
So for the Restproxy configuration on the compute nodes, does it matter if it's defined in the configuration file? For example can we get away with dummy data? The problem stems from the fact it's required in the puppet class, but we can get around that by just providing an empty string. So if it's ok for the compute nodes for the configuration to be defined as an empty string in the configuration file we can easily get around that.


As for the ivs_enabled and nfvswitch_enabled, those will also need to get added to the upstream neutron::agents::bigswitch class in puppet-neutron. If they are included in the packaging and we just need to add services, you can propose a change upstream and we can attempt a backport.

https://github.com/openstack/puppet-neutron/blob/master/manifests/agents/bigswitch.pp#L63-L75

Comment 37 Alex Schultz 2017-03-15 18:30:13 UTC
Please take a look at the proposed upstream change https://review.openstack.org/#/c/446029/ I have added in a work around for the compute rest configuration if my previous statements around the dummy data were ok.

Comment 38 bigswitch 2017-03-15 20:44:05 UTC
Alex, yes dummy(empty data) would be fine. Thanks again!

I was doing some digging around to see why things worked in Mitaka/Liberty but not in Newton. And with your guidance as to 'import neutron::agents::bigswitch' being the missing piece, I observe the following:

In Mitaka, I see the file:
https://github.com/openstack/tripleo-heat-templates/blob/stable/mitaka/puppet/manifests/overcloud_controller.pp#L432-L435

Which does the necessary imports and things worked.

In Newton (and later), this file itself seems to have been removed altogether. This seems to be missing piece. I believe we (Bigswitch) were missed out in the transition/change.

Comment 39 bigswitch 2017-03-15 20:58:33 UTC
The commit https://github.com/openstack/tripleo-heat-templates/commit/2a6ae1bab53a9dc051d9b307da3ee8d8e8c2ebc3#diff-846018e2aa141a5eb79a64b4015fc5f3
appears to have broken the integration.

Any way we could have a Composable Neutron Core Compute Plugin as well ( OS::TripleO::Services::NeutronCorePlugin) ?

Comment 40 Alex Schultz 2017-03-15 21:17:04 UTC
So the lines from the overcloud_controller didn't get transfered to the tripleo ml2 profile https://github.com/openstack/puppet-tripleo/blob/master/manifests/profile/base/neutron/plugins/ml2.pp#L63-L65 which is how this happened. I think it was because during the update the agent wasn't assumed to be part of the ml2 configuration (and upstream didn't really have a way to check). This is the bit that the new bigswitch-agent.yaml should cover. 

I've also updated my patch to include a OS::TripleO::Services::ComputeNeutronCorePlugin: OS::Heat::None in the neutron-ml2-bigswitch.yaml. I hope that's what you were asking for. Let me know if it's not

Comment 41 bigswitch 2017-03-16 19:39:23 UTC
Updating the BZ from comment on the ReviewRequest:
------------

Alex, we tested this patch in-house and the deployment failed. We found that the following changes needed to be made for a successful deployment

- puppet/extraconfig/pre_deploy/compute/neutron-ml2-bigswitch.yaml
  Looks like values for these can't be empty/none:
   neutron::plugins::ml2::bigswitch::restproxy::servers: ''
   neutron::plugins::ml2::bigswitch::restproxy::server_auth: ''
  We removed these entries (but maybe Dummy Data would be sufficient?, will try next)

- environments/neutron-ml2-bigswitch.yaml
  This line needs to be removed (my bad)
  OS::TripleO::Services::ComputeNeutronCorePlugin: OS::Heat::None

- Additionally, we had to add Compute; Service::NeutronBigswitchAgent to the roles_data.yaml file for the deployment to succeed (without this, the agent's wouldn't start on Compute nodes).

Comment 42 Alex Schultz 2017-03-16 20:13:29 UTC
You can try using '<SERVICE DEFAULT>' instead of ''. This is a hack which causes the underlying puppet module to just not define the configuration item. I updated the upstream change with this for now. I'm not sure we want add OS::TripleO::Service::NeutronBigswitchAgent to the default roles_data.yaml as that will result in the bigswitch packaging being installed by default. I think that should be a step in the installation guide for installing with bigswitch.  The other items can be added and backported with no disruption to the main product so I think we're ok to get those in and backported.

Comment 43 Alex Schultz 2017-03-16 20:16:19 UTC
There is some future work that could make it easier to enable bigswitch support with updates to puppet-tripleo, puppet-neutron and THT but right now my goal with the upstream patch is to localize the changed required that should work with 10 and 11. If we are able to merge the THT page and backport that, then you should be OK provided that the documented roles_data.yaml items are applied. I have some recommendations on better ways to integrate the agent configuration which we can address later.

Comment 44 bigswitch 2017-03-16 21:03:00 UTC
Agree with roles_data , default adding would be problem. It is on our part to have the documentation to add services into roles_data , unless there some other method to have in the environment file to override the default value 

Default value being heat-none

Comment 45 Alex Schultz 2017-03-16 21:46:58 UTC
Ah you make an excellent point about the OS::Heat::None thing. I've updated the upstream change with that.  Let me know if that works out.

Comment 46 bigswitch 2017-03-17 04:53:17 UTC
Hi Alex

We did deploy using following setting in openstack/tripleo-heat-templates / overcloud-resource-registry-puppet.j2.yaml

OS::TripleO::Services::NeutronBigswitchAgent: OS::Heat::None

use big switch environment file to override the setting ,How ever deployment did not succeed. 

For now in order deployment to be successful and big switch agent to be running following are the setting.

in environment/bigswitch.yaml 

  OS::TripleO::Services::BigswitchAgent: /usr/share/openstack-tripleo-heat-templates/puppet/services/bigswitch-agent.yaml
  OS::TripleO::Services::ComputeNeutronOvsAgent: OS::Heat::None
  OS::TripleO::Services::NeutronL3Agent: OS::Heat::None

and rest of the rest proxy and agent settings

In /puppet/extraconfig/pre_deploy/compute/neutron-ml2-bigswitch.yaml - 

             mapped_data:
                neutron::agents::bigswitch::agent_enabled: {get_input: neutron_enable_bigswitch_agent}
                neutron::agents::bigswitch::lldp_enabled: {get_input: neutron_enable_bigswitch_lldp}
                neutron::plugins::ml2::bigswitch::restproxy::servers: '<SERVICE DEFAULT>'
                neutron::plugins::ml2::bigswitch::restproxy::server_auth: '<SERVICE DEFAULT>'

In /roles_data.yaml adding following

    - OS::TripleO::Services::BigswitchAgent

Comment 47 Alex Schultz 2017-03-17 21:21:05 UTC
Ok I've updated the upstream patch to match this. Let me know if there are further issues.

Comment 48 bigswitch 2017-04-07 18:28:55 UTC
Hey Alex, we tried deploying the PR https://review.openstack.org/#/c/452278 against RHOSP10 and observed it just gets stuck in ControllerNodePreConfig step and Deployment fails (times-out). We observed that there is a difference in the file  puppet/extraconfig/pre_deploy/compute/neutron-ml2-bigswitch.yaml and puppet/extraconfig/pre_deploy/controller/neutron-ml2-bigswitch.yaml; when compared to the config that was tested in house: (diffs marked with <<--)

Working YAML:
resources:
  NeutronBigswitchConfig:
    type: OS::Heat::StructuredConfig
    properties:
      group: os-apply-config         <<-------------------
      config:
        hiera:                       <<------------------ (not present in PR)
          datafiles:
            neutron_bigswitch_data:
              mapped_data:

Not Working (https://review.openstack.org/#/c/452278):
resources:
  NeutronBigswitchConfig:
    type: OS::Heat::StructuredConfig
    properties:
      group: hiera                  <<-------------------
      config:
        datafiles:
          neutron_bigswitch_data:
            mapped_data:

I can't explain why the latter doesn't work(These changes have been highlighted in https://review.openstack.org/#/c/452377/)

Comment 49 Alex Schultz 2017-04-07 21:34:00 UTC
The interface changed between 10 and 11. Wwe've backported to Ocata (what 11 will be based off of), in order to backport to newton we need to adjust it a bit.  In 10 there is no group: hiera as that is new for 11.

Comment 50 Alex Schultz 2017-04-07 21:41:51 UTC
https://review.openstack.org/#/c/452377/ is the change for 10 and we're tracking that backport in BZ#1438893

Comment 51 bigswitch 2017-04-07 22:39:57 UTC
Thank you Alex for the clarification and confirmation. Much appreciated!

Comment 54 bigswitch 2017-08-16 16:03:42 UTC
All changes are in the upstream RH repo , Latest RHOSP 10 has all the changes in. THanks

Comment 55 bigswitch 2017-08-16 16:03:59 UTC
All changes are in the upstream RH repo , Latest RHOSP 10 has all the changes in. THanks


Note You need to log in before you can comment on or make changes to this bug.