Description of problem: We are using RHOSP7 beta2 to install a 3-controller cluster using vlan to isolate tenant traffic. The specific cmd we are using is as following [stack@manager ~]$ openstack overcloud deploy -e /home/stack/network-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml --bm-network-gateway 193.168.2.1 --floating-ip-cidr 192.168.2.0/24 --floating-ip-start 192.168.2.60 --floating-ip-end 192.168.2.90 --neutron-bridge-mappings datacentre:br-ex,vlan:br-bond --neutron-network-type vlan --neutron-network-vlan-ranges vlan:50:90 --neutron-disable-tunneling --compute-scale 1 --control-scale 3 --ceph-storage-scale 0 --plan-uuid 7fa9507f-8e9b-4567-b1fc-e883efe89c3e --debug The installation went fine and "heat resource-list overcloud" shows that all resources are at CREATE_COMPLETE status, except that in the end of overcloud installation, we got following DEBUG: neutronclient.v2_0.client Error message: {"NeutronError": {"message": "Invalid input for operation: Gateway is not valid on subnet.", "type": "InvalidInput", "detail": ""}} ERROR: openstack Invalid input for operation: Gateway is not valid on subnet. Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cliff/app.py", line 295, in run_subcommand result = cmd.run(parsed_args) File "/usr/lib/python2.7/site-packages/cliff/command.py", line 53, in run self.take_action(parsed_args) File "/usr/lib/python2.7/site-packages/rdomanager_oscplugin/v1/overcloud_deploy.py", line 678, in take_action self._deploy_postconfig(stack, parsed_args) File "/usr/lib/python2.7/site-packages/rdomanager_oscplugin/v1/overcloud_deploy.py", line 557, in _deploy_postconfig keystone_client=keystone_client, File "/usr/lib/python2.7/site-packages/os_cloud_config/neutron.py", line 52, in initialize_neutron _create_subnet(neutron_client, ext_net, network_desc, 'external', None) File "/usr/lib/python2.7/site-packages/os_cloud_config/neutron.py", line 137, in _create_subnet return neutron.create_subnet({'subnet': subnet}) File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 102, in with_params ret = self.function(instance, *args, **kwargs) File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 597, in create_subnet return self.post(self.subnets_path, body=body) File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 298, in post headers=headers, params=params) File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 211, in do_request self._handle_fault_response(status_code, replybody) File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 185, in _handle_fault_response exception_handler_v20(status_code, des_error_body) File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 70, in exception_handler_v20 status_code=status_code) BadRequest: Invalid input for operation: Gateway is not valid on subnet. DEBUG: openstackclient.shell clean_up DeployOvercloud DEBUG: openstackclient.shell got an error: Invalid input for operation: Gateway is not valid on subnet. ERROR: openstackclient.shell Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/openstackclient/shell.py", line 176, in run return super(OpenStackShell, self).run(argv) File "/usr/lib/python2.7/site-packages/cliff/app.py", line 230, in run result = self.run_subcommand(remainder) File "/usr/lib/python2.7/site-packages/cliff/app.py", line 295, in run_subcommand result = cmd.run(parsed_args) File "/usr/lib/python2.7/site-packages/cliff/command.py", line 53, in run self.take_action(parsed_args) File "/usr/lib/python2.7/site-packages/rdomanager_oscplugin/v1/overcloud_deploy.py", line 678, in take_action self._deploy_postconfig(stack, parsed_args) File "/usr/lib/python2.7/site-packages/rdomanager_oscplugin/v1/overcloud_deploy.py", line 557, in _deploy_postconfig keystone_client=keystone_client, File "/usr/lib/python2.7/site-packages/os_cloud_config/neutron.py", line 52, in initialize_neutron _create_subnet(neutron_client, ext_net, network_desc, 'external', None) File "/usr/lib/python2.7/site-packages/os_cloud_config/neutron.py", line 137, in _create_subnet return neutron.create_subnet({'subnet': subnet}) File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 102, in with_params ret = self.function(instance, *args, **kwargs) File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 597, in create_subnet return self.post(self.subnets_path, body=body) File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 298, in post headers=headers, params=params) File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 211, in do_request self._handle_fault_response(status_code, replybody) File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 185, in _handle_fault_response exception_handler_v20(status_code, des_error_body) File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 70, in exception_handler_v20 status_code=status_code) BadRequest: Invalid input for operation: Gateway is not valid on subnet. After installation, we found cluster is not running on any of the 3 controller nodes #pcs status #Error: cluster is not currently running on this node Also neutron services are not properly running [stack@manager ~]$ source overcloudrc [stack@manager ~]$ neutron agent-list +--------------------------------------+--------------------+------------------------------------+-------+----------------+---------------------------+ | id | agent_type | host | alive | admin_state_up | binary | +--------------------------------------+--------------------+------------------------------------+-------+----------------+---------------------------+ | 26caf01b-75db-49c6-bd7f-9fba512a3eb8 | Metadata agent | overcloud-controller-2.localdomain | xxx | True | neutron-metadata-agent | | 5a1a9d85-5e2b-4c7c-b96e-198c597ab18a | Metadata agent | overcloud-controller-0.localdomain | xxx | True | neutron-metadata-agent | | 5c7b948c-3901-4eb4-a5c6-4c32a2748c25 | Open vSwitch agent | neutron-n-1 | xxx | True | neutron-openvswitch-agent | | 6f9433b4-de37-430b-b0eb-93bfdac8ba48 | Open vSwitch agent | neutron-n-2 | xxx | True | neutron-openvswitch-agent | | 740f6601-3776-4c18-944d-1534aeb22533 | L3 agent | overcloud-controller-1.localdomain | xxx | True | neutron-l3-agent | | ad926177-bd8d-4b56-a53a-976700534d10 | Metadata agent | overcloud-controller-1.localdomain | xxx | True | neutron-metadata-agent | | c20507ca-60fd-4ad5-93e9-86ff3c96e0e4 | DHCP agent | neutron-n-0 | xxx | True | neutron-dhcp-agent | | cd54366a-e672-4961-8328-4ad8d951753c | Open vSwitch agent | neutron-n-0 | xxx | True | neutron-openvswitch-agent | | cefd81b7-8988-4494-9294-6b5515edab30 | DHCP agent | neutron-n-1 | xxx | True | neutron-dhcp-agent | | d399d5be-d368-441e-9e7b-9b8ce6077815 | L3 agent | overcloud-controller-0.localdomain | xxx | True | neutron-l3-agent | | db9685dc-0dbe-4362-83bc-5af3d3d50de2 | L3 agent | overcloud-controller-2.localdomain | xxx | True | neutron-l3-agent | | eb9375d6-d98b-43ed-bd99-0b9270001de8 | DHCP agent | neutron-n-2 | xxx | True | neutron-dhcp-agent | | eee301e8-694e-48d6-9160-e976ebb5000a | Open vSwitch agent | overcloud-compute-0.localdomain | :-) | True | neutron-openvswitch-agent | +--------------------------------------+--------------------+------------------------------------+-------+----------------+---------------------------+ Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
We we told to try following cmds to make HA cluster and neutron agents work properly. However, they didn't help much. [root@overcloud-controller-1 heat-admin]# pcs constraint order remove neutron-ovs-cleanup-clone [root@overcloud-controller-1 heat-admin]# pcs constraint order remove colocation Error: No matching resources found in ordering list [root@overcloud-controller-1 heat-admin]# crm_resource -C --resource neutron-netns-cleanup-clone Cleaning up neutron-netns-cleanup:0 on overcloud-controller-0 Cleaning up neutron-netns-cleanup:0 on overcloud-controller-1 Cleaning up neutron-netns-cleanup:0 on overcloud-controller-2 Cleaning up neutron-netns-cleanup:1 on overcloud-controller-0 Cleaning up neutron-netns-cleanup:1 on overcloud-controller-1 Cleaning up neutron-netns-cleanup:1 on overcloud-controller-2 Cleaning up neutron-netns-cleanup:2 on overcloud-controller-0 Cleaning up neutron-netns-cleanup:2 on overcloud-controller-1 Cleaning up neutron-netns-cleanup:2 on overcloud-controller-2 Waiting for 9 replies from the CRMd......... OK [root@overcloud-controller-1 heat-admin]# openstack-service restart neutron Too few arguments.
looks this was accidentally opened twice. closing this one *** This bug has been marked as a duplicate of bug 1241793 ***