Bug 1266183

Summary:	neutron services is not running on one controller node
Product:	Red Hat OpenStack	Reporter:	bigswitch <rhosp-bugs-internal>
Component:	rhosp-director	Assignee:	Marios Andreou <mandreou>
Status:	CLOSED ERRATA	QA Contact:	Udi Shkalim <ushkalim>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	7.0 (Kilo)	CC:	ebarrera, hbrock, mburns, oblaut, rhel-osp-director-maint, rhosp-bugs-internal, saime
Target Milestone:	---	Keywords:	TestOnly, Triaged
Target Release:	8.0 (Liberty)
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2016-04-07 21:40:24 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description bigswitch 2015-09-24 17:32:47 UTC

Description of problem:
Did a new deployment with three controller and one compute. after deployment is successful, notice some neutron services on overcloud-control-2 is not running. manually restart the services work

[stack@dell-undercloud ~]$ heat stack-list
+--------------------------------------+------------+-----------------+----------------------+
| id                                   | stack_name | stack_status    | creation_time        |
+--------------------------------------+------------+-----------------+----------------------+
| 22a91797-dd4a-451e-927c-4bc5990ff471 | overcloud  | CREATE_COMPLETE | 2015-09-24T16:18:32Z |
+--------------------------------------+------------+-----------------+----------------------+
[stack@dell-undercloud ~]$ source overcloudrc
[stack@dell-undercloud ~]$ neutron agent-list
+--------------------------------------+--------------------+------------------------------------+-------+----------------+---------------------------+
| id                                   | agent_type         | host                               | alive | admin_state_up | binary                    |
+--------------------------------------+--------------------+------------------------------------+-------+----------------+---------------------------+
| 126eb7ba-8eff-4599-a71f-62b23e992379 | Open vSwitch agent | overcloud-controller-0.localdomain | :-)   | True           | neutron-openvswitch-agent |
| 2dac3402-cbc3-4903-b0ca-be626eda4b01 | Open vSwitch agent | overcloud-compute-0.localdomain    | :-)   | True           | neutron-openvswitch-agent |
| 3cadc9dc-d561-4553-ac6d-dfd2a4a6fa07 | Metadata agent     | overcloud-controller-0.localdomain | :-)   | True           | neutron-metadata-agent    |
| 4129e624-df06-4350-bd60-3c7840a2ea3a | Open vSwitch agent | overcloud-controller-1.localdomain | :-)   | True           | neutron-openvswitch-agent |
| 519e626f-0a09-4cbe-9730-81f2bdc97eb5 | Metadata agent     | overcloud-controller-2.localdomain | xxx   | True           | neutron-metadata-agent    |
| 6566e981-ea12-4bd8-a8f4-46d7575a816b | DHCP agent         | overcloud-controller-2.localdomain | xxx   | True           | neutron-dhcp-agent        |
| 66db8bf8-59c9-4a1e-a671-0c15acd9501f | L3 agent           | overcloud-controller-0.localdomain | :-)   | True           | neutron-l3-agent          |
| 6f47e13c-e6d5-4fe0-911c-7ab0d81fdfc5 | L3 agent           | overcloud-controller-1.localdomain | :-)   | True           | neutron-l3-agent          |
| 90865d7e-db52-4fe1-8801-f7e159f9f4dc | DHCP agent         | overcloud-controller-0.localdomain | :-)   | True           | neutron-dhcp-agent        |
| b3fffb3b-fd83-4f61-ad02-a3085e9e6658 | Open vSwitch agent | overcloud-controller-2.localdomain | :-)   | True           | neutron-openvswitch-agent |
| bb89dc91-4154-4d3a-bf9d-efdadfa98658 | Metadata agent     | overcloud-controller-1.localdomain | :-)   | True           | neutron-metadata-agent    |
| bec80895-d553-43ef-975f-5cf3c4545c26 | L3 agent           | overcloud-controller-2.localdomain | xxx   | True           | neutron-l3-agent          |
| d541ef16-8b42-4799-b2e7-d3c3f17f5502 | DHCP agent         | overcloud-controller-1.localdomain | :-)   | True           | neutron-dhcp-agent        |
| ff418ec1-f807-44a9-aea4-47a5345bad51 | Open vSwitch agent | overcloud-compute-35.localdomain   | :-)   | True           | neutron-openvswitch-agent |
+--------------------------------------+--------------------+------------------------------------+-------+----------------+---------------------------+
[stack@dell-undercloud ~]$ 

[heat-admin@overcloud-controller-2 ~]$ systemctl -a | grep neutron
neutron-bsn-lldp.service                                                                                         loaded active   running   bsn lldp
neutron-dhcp-agent.service                                                                                       loaded failed   failed    OpenStack Neutron DHCP Agent
neutron-openvswitch-agent.service                                                                                loaded active   running   Cluster Controlled neutron-openvswitch-agent
neutron-server.service                                                                                           loaded active   running   Cluster Controlled neutron-server

[stack@dell-undercloud ~]$ neutron agent-list
+--------------------------------------+--------------------+------------------------------------+-------+----------------+---------------------------+
| id                                   | agent_type         | host                               | alive | admin_state_up | binary                    |
+--------------------------------------+--------------------+------------------------------------+-------+----------------+---------------------------+
| 126eb7ba-8eff-4599-a71f-62b23e992379 | Open vSwitch agent | overcloud-controller-0.localdomain | :-)   | True           | neutron-openvswitch-agent |
| 2dac3402-cbc3-4903-b0ca-be626eda4b01 | Open vSwitch agent | overcloud-compute-0.localdomain    | :-)   | True           | neutron-openvswitch-agent |
| 3cadc9dc-d561-4553-ac6d-dfd2a4a6fa07 | Metadata agent     | overcloud-controller-0.localdomain | :-)   | True           | neutron-metadata-agent    |
| 4129e624-df06-4350-bd60-3c7840a2ea3a | Open vSwitch agent | overcloud-controller-1.localdomain | :-)   | True           | neutron-openvswitch-agent |
| 519e626f-0a09-4cbe-9730-81f2bdc97eb5 | Metadata agent     | overcloud-controller-2.localdomain | :-)   | True           | neutron-metadata-agent    |
| 6566e981-ea12-4bd8-a8f4-46d7575a816b | DHCP agent         | overcloud-controller-2.localdomain | :-)   | True           | neutron-dhcp-agent        |
| 66db8bf8-59c9-4a1e-a671-0c15acd9501f | L3 agent           | overcloud-controller-0.localdomain | :-)   | True           | neutron-l3-agent          |
| 6f47e13c-e6d5-4fe0-911c-7ab0d81fdfc5 | L3 agent           | overcloud-controller-1.localdomain | :-)   | True           | neutron-l3-agent          |
| 90865d7e-db52-4fe1-8801-f7e159f9f4dc | DHCP agent         | overcloud-controller-0.localdomain | :-)   | True           | neutron-dhcp-agent        |
| b3fffb3b-fd83-4f61-ad02-a3085e9e6658 | Open vSwitch agent | overcloud-controller-2.localdomain | :-)   | True           | neutron-openvswitch-agent |
| bb89dc91-4154-4d3a-bf9d-efdadfa98658 | Metadata agent     | overcloud-controller-1.localdomain | :-)   | True           | neutron-metadata-agent    |
| bec80895-d553-43ef-975f-5cf3c4545c26 | L3 agent           | overcloud-controller-2.localdomain | :-)   | True           | neutron-l3-agent          |
| d541ef16-8b42-4799-b2e7-d3c3f17f5502 | DHCP agent         | overcloud-controller-1.localdomain | :-)   | True           | neutron-dhcp-agent        |
| ff418ec1-f807-44a9-aea4-47a5345bad51 | Open vSwitch agent | overcloud-compute-35.localdomain   | :-)   | True           | neutron-openvswitch-agent |
+--------------------------------------+--------------------+------------------------------------+-------+----------------+---------------------------+

Version-Release number of selected component (if applicable):


How reproducible:
seen it once

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 bigswitch 2015-09-24 17:34:29 UTC

sosreport from undercloud as well as overcloud-control-2 is at

https://bigswitch.box.com/s/gm9ajwsiaayygyjhdb7a83a8jh74mspg

Comment 3 chris alfonso 2015-09-24 17:53:11 UTC

Did a pcs resource cleanup do anything to resolve the service availability?

Comment 11 Udi Shkalim 2016-01-11 18:29:08 UTC

Verified on 7.2 - 2015.1.2

After overcloud deployment - virt environment 3 controllers 1 compute

[root@overcloud-controller-2 ~]# pcs status
Cluster name: tripleo_cluster
Last updated: Mon Jan 11 13:27:57 2016		Last change: Mon Jan 11 11:35:15 2016 by root via crm_resource on overcloud-controller-0
Stack: corosync
Current DC: overcloud-controller-2 (version 1.1.13-10.el7-44eb2dd) - partition with quorum
3 nodes and 108 resources configured

Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]

Full list of resources:

 ip-192.0.2.6	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-0
 Clone Set: haproxy-clone [haproxy]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 ip-192.0.2.7	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-1
 Master/Slave Set: galera-master [galera]
     Masters: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Master/Slave Set: redis-master [redis]
     Masters: [ overcloud-controller-1 ]
     Slaves: [ overcloud-controller-0 overcloud-controller-2 ]
 Clone Set: mongod-clone [mongod]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: rabbitmq-clone [rabbitmq]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: memcached-clone [memcached]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-nova-scheduler-clone [openstack-nova-scheduler]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-l3-agent-clone [neutron-l3-agent]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-ceilometer-alarm-notifier-clone [openstack-ceilometer-alarm-notifier]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-heat-engine-clone [openstack-heat-engine]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-ceilometer-api-clone [openstack-ceilometer-api]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-metadata-agent-clone [neutron-metadata-agent]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-ovs-cleanup-clone [neutron-ovs-cleanup]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-netns-cleanup-clone [neutron-netns-cleanup]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-heat-api-clone [openstack-heat-api]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-cinder-scheduler-clone [openstack-cinder-scheduler]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-nova-api-clone [openstack-nova-api]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-heat-api-cloudwatch-clone [openstack-heat-api-cloudwatch]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-ceilometer-collector-clone [openstack-ceilometer-collector]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-keystone-clone [openstack-keystone]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-nova-consoleauth-clone [openstack-nova-consoleauth]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-glance-registry-clone [openstack-glance-registry]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-ceilometer-notification-clone [openstack-ceilometer-notification]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-cinder-api-clone [openstack-cinder-api]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-dhcp-agent-clone [neutron-dhcp-agent]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-glance-api-clone [openstack-glance-api]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-openvswitch-agent-clone [neutron-openvswitch-agent]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-nova-novncproxy-clone [openstack-nova-novncproxy]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: delay-clone [delay]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-server-clone [neutron-server]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: httpd-clone [httpd]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-ceilometer-central-clone [openstack-ceilometer-central]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-ceilometer-alarm-evaluator-clone [openstack-ceilometer-alarm-evaluator]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: openstack-heat-api-cfn-clone [openstack-heat-api-cfn]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 openstack-cinder-volume	(systemd:openstack-cinder-volume):	Started overcloud-controller-2
 Clone Set: openstack-nova-conductor-clone [openstack-nova-conductor]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]

PCSD Status:
  overcloud-controller-0: Online
  overcloud-controller-1: Online
  overcloud-controller-2: Online

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled

 
[stack@instack ~]$ neutron agent-list
+--------------------------------------+--------------------+------------------------------------+-------+----------------+---------------------------+
| id                                   | agent_type         | host                               | alive | admin_state_up | binary                    |
+--------------------------------------+--------------------+------------------------------------+-------+----------------+---------------------------+
| 13ab282f-2405-44e8-b790-defa5db3ac08 | L3 agent           | overcloud-controller-2.localdomain | :-)   | True           | neutron-l3-agent          |
| 3e777156-f477-4e63-b6ff-538201774b61 | Metadata agent     | overcloud-controller-0.localdomain | :-)   | True           | neutron-metadata-agent    |
| 55412dad-5305-4763-a219-8b52071d726e | DHCP agent         | overcloud-controller-1.localdomain | :-)   | True           | neutron-dhcp-agent        |
| 707b992d-8856-4d7d-8da6-6f134b06c1a9 | Open vSwitch agent | overcloud-controller-1.localdomain | :-)   | True           | neutron-openvswitch-agent |
| 7a8a7d09-d593-4d3b-a71e-b34f63a39570 | Open vSwitch agent | overcloud-controller-0.localdomain | :-)   | True           | neutron-openvswitch-agent |
| 83b4a90a-023a-45c2-908f-7e67f190eebd | Metadata agent     | overcloud-controller-2.localdomain | :-)   | True           | neutron-metadata-agent    |
| a1724073-a211-41ab-b64e-362591a49ead | L3 agent           | overcloud-controller-0.localdomain | :-)   | True           | neutron-l3-agent          |
| a6c8baf3-86f6-4c6f-a2d8-04aed9c3b8d1 | Open vSwitch agent | overcloud-compute-0.localdomain    | :-)   | True           | neutron-openvswitch-agent |
| b4367e87-1cd6-491d-a318-1826592378b0 | Open vSwitch agent | overcloud-controller-2.localdomain | :-)   | True           | neutron-openvswitch-agent |
| c96831fe-fa71-4275-b0f4-b7066c34ebe4 | Metadata agent     | overcloud-controller-1.localdomain | :-)   | True           | neutron-metadata-agent    |
| dafc6340-6534-4e62-a066-37439c001446 | DHCP agent         | overcloud-controller-0.localdomain | :-)   | True           | neutron-dhcp-agent        |
| e55627c3-f803-45cf-a490-13113d7a6fff | DHCP agent         | overcloud-controller-2.localdomain | :-)   | True           | neutron-dhcp-agent        |
| fcb21af6-d01d-42fc-8b33-873344410809 | L3 agent           | overcloud-controller-1.localdomain | :-)   | True           | neutron-l3-agent          |
+--------------------------------------+--------------------+------------------------------------+-------+----------------+---------------------------+

Comment 16 errata-xmlrpc 2016-04-07 21:40:24 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-0604.html