Bug 1241801 - HA cluster is not properly formed in a 3-controller deployment
Summary: HA cluster is not properly formed in a 3-controller deployment
Keywords:
Status: CLOSED DUPLICATE of bug 1241793
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: rhosp-director
Version: 7.0 (Kilo)
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: Director
Assignee: chris alfonso
QA Contact: yeylon@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-07-10 06:49 UTC by bigswitch
Modified: 2016-04-18 07:13 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-07-10 11:32:17 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description bigswitch 2015-07-10 06:49:17 UTC
Description of problem:
We are using RHOSP7 beta2 to install a 3-controller cluster using vlan to isolate tenant traffic. The specific cmd we are using is as following

[stack@manager ~]$ openstack overcloud deploy -e /home/stack/network-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml --bm-network-gateway 193.168.2.1 --floating-ip-cidr 192.168.2.0/24 --floating-ip-start 192.168.2.60 --floating-ip-end 192.168.2.90 --neutron-bridge-mappings datacentre:br-ex,vlan:br-bond --neutron-network-type vlan --neutron-network-vlan-ranges vlan:50:90 --neutron-disable-tunneling --compute-scale 1 --control-scale 3 --ceph-storage-scale 0 --plan-uuid 7fa9507f-8e9b-4567-b1fc-e883efe89c3e --debug

The installation went fine and "heat resource-list overcloud" shows that all resources are at CREATE_COMPLETE status, except that in the end of overcloud installation, we got following

DEBUG: neutronclient.v2_0.client Error message: {"NeutronError": {"message": "Invalid input for operation: Gateway is not valid on subnet.", "type": "InvalidInput", "detail": ""}}
ERROR: openstack Invalid input for operation: Gateway is not valid on subnet.
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 295, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 53, in run
    self.take_action(parsed_args)
  File "/usr/lib/python2.7/site-packages/rdomanager_oscplugin/v1/overcloud_deploy.py", line 678, in take_action
    self._deploy_postconfig(stack, parsed_args)
  File "/usr/lib/python2.7/site-packages/rdomanager_oscplugin/v1/overcloud_deploy.py", line 557, in _deploy_postconfig
    keystone_client=keystone_client,
  File "/usr/lib/python2.7/site-packages/os_cloud_config/neutron.py", line 52, in initialize_neutron
    _create_subnet(neutron_client, ext_net, network_desc, 'external', None)
  File "/usr/lib/python2.7/site-packages/os_cloud_config/neutron.py", line 137, in _create_subnet
    return neutron.create_subnet({'subnet': subnet})
  File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 102, in with_params
    ret = self.function(instance, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 597, in create_subnet
    return self.post(self.subnets_path, body=body)
  File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 298, in post
    headers=headers, params=params)
  File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 211, in do_request
    self._handle_fault_response(status_code, replybody)
  File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 185, in _handle_fault_response
    exception_handler_v20(status_code, des_error_body)
  File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 70, in exception_handler_v20
    status_code=status_code)
BadRequest: Invalid input for operation: Gateway is not valid on subnet.
DEBUG: openstackclient.shell clean_up DeployOvercloud
DEBUG: openstackclient.shell got an error: Invalid input for operation: Gateway is not valid on subnet.
ERROR: openstackclient.shell Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/openstackclient/shell.py", line 176, in run
    return super(OpenStackShell, self).run(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 230, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 295, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 53, in run
    self.take_action(parsed_args)
  File "/usr/lib/python2.7/site-packages/rdomanager_oscplugin/v1/overcloud_deploy.py", line 678, in take_action
    self._deploy_postconfig(stack, parsed_args)
  File "/usr/lib/python2.7/site-packages/rdomanager_oscplugin/v1/overcloud_deploy.py", line 557, in _deploy_postconfig
    keystone_client=keystone_client,
  File "/usr/lib/python2.7/site-packages/os_cloud_config/neutron.py", line 52, in initialize_neutron
    _create_subnet(neutron_client, ext_net, network_desc, 'external', None)
  File "/usr/lib/python2.7/site-packages/os_cloud_config/neutron.py", line 137, in _create_subnet
    return neutron.create_subnet({'subnet': subnet})
  File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 102, in with_params
    ret = self.function(instance, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 597, in create_subnet
    return self.post(self.subnets_path, body=body)
  File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 298, in post
    headers=headers, params=params)
  File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 211, in do_request
    self._handle_fault_response(status_code, replybody)
  File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 185, in _handle_fault_response
    exception_handler_v20(status_code, des_error_body)
  File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 70, in exception_handler_v20
    status_code=status_code)
BadRequest: Invalid input for operation: Gateway is not valid on subnet.

After installation, we found cluster is not running on any of the 3 controller nodes
#pcs status
#Error: cluster is not currently running on this node

Also neutron services are not properly running
[stack@manager ~]$ source overcloudrc 

[stack@manager ~]$ neutron agent-list

+--------------------------------------+--------------------+------------------------------------+-------+----------------+---------------------------+

| id                                   | agent_type         | host                               | alive | admin_state_up | binary                    |

+--------------------------------------+--------------------+------------------------------------+-------+----------------+---------------------------+

| 26caf01b-75db-49c6-bd7f-9fba512a3eb8 | Metadata agent     | overcloud-controller-2.localdomain | xxx   | True           | neutron-metadata-agent    |

| 5a1a9d85-5e2b-4c7c-b96e-198c597ab18a | Metadata agent     | overcloud-controller-0.localdomain | xxx   | True           | neutron-metadata-agent    |

| 5c7b948c-3901-4eb4-a5c6-4c32a2748c25 | Open vSwitch agent | neutron-n-1                        | xxx   | True           | neutron-openvswitch-agent |

| 6f9433b4-de37-430b-b0eb-93bfdac8ba48 | Open vSwitch agent | neutron-n-2                        | xxx   | True           | neutron-openvswitch-agent |

| 740f6601-3776-4c18-944d-1534aeb22533 | L3 agent           | overcloud-controller-1.localdomain | xxx   | True           | neutron-l3-agent          |

| ad926177-bd8d-4b56-a53a-976700534d10 | Metadata agent     | overcloud-controller-1.localdomain | xxx   | True           | neutron-metadata-agent    |

| c20507ca-60fd-4ad5-93e9-86ff3c96e0e4 | DHCP agent         | neutron-n-0                        | xxx   | True           | neutron-dhcp-agent        |

| cd54366a-e672-4961-8328-4ad8d951753c | Open vSwitch agent | neutron-n-0                        | xxx   | True           | neutron-openvswitch-agent |

| cefd81b7-8988-4494-9294-6b5515edab30 | DHCP agent         | neutron-n-1                        | xxx   | True           | neutron-dhcp-agent        |

| d399d5be-d368-441e-9e7b-9b8ce6077815 | L3 agent           | overcloud-controller-0.localdomain | xxx   | True           | neutron-l3-agent          |

| db9685dc-0dbe-4362-83bc-5af3d3d50de2 | L3 agent           | overcloud-controller-2.localdomain | xxx   | True           | neutron-l3-agent          |

| eb9375d6-d98b-43ed-bd99-0b9270001de8 | DHCP agent         | neutron-n-2                        | xxx   | True           | neutron-dhcp-agent        |

| eee301e8-694e-48d6-9160-e976ebb5000a | Open vSwitch agent | overcloud-compute-0.localdomain    | :-)   | True           | neutron-openvswitch-agent |

+--------------------------------------+--------------------+------------------------------------+-------+----------------+---------------------------+


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 3 bigswitch 2015-07-10 07:09:10 UTC
We we told to try following cmds to make HA cluster and neutron agents work properly. However, they didn't help much. 

[root@overcloud-controller-1 heat-admin]# pcs constraint order remove neutron-ovs-cleanup-clone
[root@overcloud-controller-1 heat-admin]# pcs constraint order remove colocation
Error: No matching resources found in ordering list
[root@overcloud-controller-1 heat-admin]# crm_resource -C --resource neutron-netns-cleanup-clone
Cleaning up neutron-netns-cleanup:0 on overcloud-controller-0
Cleaning up neutron-netns-cleanup:0 on overcloud-controller-1
Cleaning up neutron-netns-cleanup:0 on overcloud-controller-2
Cleaning up neutron-netns-cleanup:1 on overcloud-controller-0
Cleaning up neutron-netns-cleanup:1 on overcloud-controller-1
Cleaning up neutron-netns-cleanup:1 on overcloud-controller-2
Cleaning up neutron-netns-cleanup:2 on overcloud-controller-0
Cleaning up neutron-netns-cleanup:2 on overcloud-controller-1
Cleaning up neutron-netns-cleanup:2 on overcloud-controller-2
Waiting for 9 replies from the CRMd......... OK
[root@overcloud-controller-1 heat-admin]# openstack-service restart neutron
Too few arguments.

Comment 4 Mike Burns 2015-07-10 11:32:17 UTC
looks this was accidentally opened twice.  closing this one

*** This bug has been marked as a duplicate of bug 1241793 ***


Note You need to log in before you can comment on or make changes to this bug.