Bug 1238750

Summary: When deploying HA some neutron services are down
Product: Red Hat OpenStack Reporter: Udi Kalifon <ukalifon>
Component: rhosp-directorAssignee: Marios Andreou <mandreou>
Status: CLOSED ERRATA QA Contact: Udi Kalifon <ukalifon>
Severity: unspecified Docs Contact:
Priority: high    
Version: DirectorCC: calfonso, dmacpher, jason.dobies, mburns, oblaut, rhel-osp-director-maint, ukalifon, yeylon
Target Milestone: gaKeywords: TestOnly, Triaged
Target Release: Director   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-0.8.6-40.el7ost python-rdomanager-oscplugin-0.0.8-37.el7ost Doc Type: Bug Fix
Doc Text:
The NeutronScale resource renamed neutron agents on Controller nodes. This caused an inconsistency with the "neutron agent-list" and as result Neutron reported errors of not having enough L3 agents for L3 HA. This fix removes the NeutronScale resource from Overcloud Heat templates and plans. NeutronScale does not appear in "neutron agent-list" and Neutron reports no errors.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2015-08-05 13:58:14 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1238117    
Bug Blocks:    
Attachments:
Description Flags
/var/log/messages from the controller node where pcs status was called
none
logs of overcloud-controller-0 none

Description Udi Kalifon 2015-07-02 14:12:26 UTC 
Description of problem:
After deploying HA, if you run "pcs status" you see some failed actions:

Failed actions:
    openstack-heat-api_monitor_60000 on overcloud-controller-0 'not running' (7): call=197, status=complete, exit-reason='none', last-rc-change='Thu Jul  2 09:14:32 2015', queued=0ms, exec=3ms
    neutron-openvswitch-agent_monitor_60000 on overcloud-controller-0 'not running' (7): call=245, status=complete, exit-reason='none', last-rc-change='Thu Jul  2 09:15:44 2015', queued=0ms, exec=0ms
    neutron-openvswitch-agent_start_0 on overcloud-controller-1 'not running' (7): call=241, status=complete, exit-reason='none', last-rc-change='Thu Jul  2 09:14:49 2015', queued=2001ms, exec=75ms


When checking with "neutron agent-list" you see some services that are down:

]$ neutron agent-list
+---------+--------------------+------------------------------------+-------+----------------+---------------------------+
| id      | agent_type         | host                               | alive | admin_state_up | binary                    |
+---------+--------------------+------------------------------------+-------+----------------+---------------------------+
| 08bc... | Metadata agent     | neutron-n-0                        | :-)   | True           | neutron-metadata-agent    |
| 1273... | DHCP agent         | overcloud-controller-0.localdomain | xxx   | True           | neutron-dhcp-agent        |
| 171b... | Metadata agent     | overcloud-controller-0.localdomain | xxx   | True           | neutron-metadata-agent    |
| 2969... | L3 agent           | overcloud-controller-2.localdomain | :-)   | True           | neutron-l3-agent          |
| 4678... | DHCP agent         | neutron-n-0                        | :-)   | True           | neutron-dhcp-agent        |
| 55d6... | DHCP agent         | overcloud-controller-2.localdomain | :-)   | True           | neutron-dhcp-agent        |
| 60f1... | Open vSwitch agent | overcloud-compute-0.localdomain    | :-)   | True           | neutron-openvswitch-agent |
| 6bee... | Open vSwitch agent | neutron-n-2                        | :-)   | True           | neutron-openvswitch-agent |
| 9c96... | Open vSwitch agent | neutron-n-0                        | :-)   | True           | neutron-openvswitch-agent |
| b1cc... | L3 agent           | overcloud-controller-0.localdomain | xxx   | True           | neutron-l3-agent          |
| d5a5... | L3 agent           | neutron-n-0                        | :-)   | True           | neutron-l3-agent          |
| f3a6... | L3 agent           | overcloud-controller-1.localdomain | xxx   | True           | neutron-l3-agent          |
| f49a... | Metadata agent     | overcloud-controller-1.localdomain | xxx   | True           | neutron-metadata-agent    |
| fe90... | Metadata agent     | overcloud-controller-2.localdomain | :-)   | True           | neutron-metadata-agent    |
+---------+--------------------+------------------------------------+-------+----------------+---------------------------+


How reproducible:
100%


Steps to Reproduce:
1. Deploy HA on bare metals
2. Log in to the controller as root, and run pcs status
3. Use the overcloudrc file and run neutron agent-list
Comment 3 chris alfonso 2015-07-02 17:10:10 UTC 
Udi, can you provide logs to detail what's happening here?
Comment 5 Marios Andreou 2015-07-03 10:31:34 UTC 
Udi, Chris, at least wrt to the neutron agent list, this looks to me like https://bugzilla.redhat.com/show_bug.cgi?id=1238117 - HOWEVER no idea what the heat-api-monitor is about. 

Udi is this on BM?
Comment 6 Udi Kalifon 2015-07-05 06:59:01 UTC 
Created attachment 1046132 [details]
/var/log/messages from the controller node where pcs status was called

Marios - yes, this is on bare metals.

Chris - I'm attaching the controller node's /var/log/messages which I hope is what you wanted (it shows the deployment of the node in the beginning of the file). This problem is easily reproducible too.
Comment 7 Marios Andreou 2015-07-06 09:26:05 UTC 
Udi I poked but can't see something useful in the messages. If it is (and looks like it from your output above) like the issue at 
https://bugzilla.redhat.com/show_bug.cgi?id=1236578 then there wouldn't be something to see there (though in your case you had enough l3 agents [neutron-n-?], but the old ones were still in intermediate state (most are xxx but some are still :-).

I think the fix at https://review.gerrithub.io/#/c/238320/ (esp v6, and depending on your/any feedback we can tweak the sleep time or other parameters) will help here. Would be great if you can test that out against this, let me know if you need help applying.

Thanks, this bug is important because until now I thought the issues with NeutronScale were confined to 'something happening too fast in a VM environment'
Comment 8 chris alfonso 2015-07-06 15:47:39 UTC 
We are getting mixed reports on this one. Omri didn't see the same behavior and we think what you're seeing might be fixed in the latest puddle.
Comment 9 Jay Dobies 2015-07-06 19:43:47 UTC 
Moving back to ON_DEV and making it depend on 1238117. This *should* be fixed by the fix for that, so this should be moved back to ON_QA along with 1238117.
Comment 10 Marios Andreou 2015-07-16 13:16:52 UTC 
confirmed today for https://bugzilla.redhat.com/show_bug.cgi?id=1238117#c26 python-rdomanager-oscplugin-0.0.8-38.el7ost.noarch includes the neutronscale removal and should fix this bug
Comment 11 Ofer Blaut 2015-07-19 11:15:16 UTC 
Hi still see it


Failed actions:
    openstack-heat-api_monitor_60000 on overcloud-controller-2 'not running' (7): call=219, status=complete, exit-reason='none', last-rc-change='Sun Jul 19 04:59:01 2015', queued=0ms, exec=0ms
    neutron-openvswitch-agent_monitor_60000 on overcloud-controller-2 'not running' (7): call=328, status=complete, exit-reason='none', last-rc-change='Sun Jul 19 05:03:25 2015', queued=0ms, exec=0ms
    openstack-heat-api_monitor_60000 on overcloud-controller-1 'not running' (7): call=221, status=complete, exit-reason='none', last-rc-change='Sun Jul 19 04:59:02 2015', queued=0ms, exec=0ms
    neutron-openvswitch-agent_monitor_60000 on overcloud-controller-1 'not running' (7): call=329, status=complete, exit-reason='none', last-rc-change='Sun Jul 19 05:03:25 2015', queued=0ms, exec=0ms
    openstack-heat-api-cloudwatch_monitor_60000 on overcloud-controller-0 'not running' (7): call=225, status=complete, exit-reason='none', last-rc-change='Sun Jul 19 04:59:02 2015', queued=0ms, exec=0ms
    neutron-openvswitch-agent_monitor_60000 on overcloud-controller-0 'not running' (7): call=352, status=complete, exit-reason='none', last-rc-change='Sun Jul 19 05:03:50 2015', queued=0ms, exec=0ms
    neutron-server_monitor_60000 on overcloud-controller-0 'not running' (7): call=346, status=complete, exit-reason='none', last-rc-change='Sun Jul 19 05:03:46 2015', queued=0ms, exec=0ms





  pacemaker: active/enabled
  pcsd: active/enabled
[root@overcloud-controller-0 heat-admin]# 
[root@overcloud-controller-0 heat-admin]# cat /var/log/neutron/server.log 
2015-07-19 04:58:02.061 22991 WARNING neutron.agent.securitygroups_rpc [-] Driver configuration doesn't match with enable_security_group
2015-07-19 04:58:02.062 22991 WARNING neutron.agent.securitygroups_rpc [-] Driver configuration doesn't match with enable_security_group
2015-07-19 04:58:04.446 22991 WARNING neutron.agent.securitygroups_rpc [-] Driver configuration doesn't match with enable_security_group
2015-07-19 04:58:04.556 22991 WARNING neutron.api.extensions [-] Extension flavor not supported by any of loaded plugins
2015-07-19 04:58:04.569 22991 WARNING neutron.api.extensions [-] Extension metering not supported by any of loaded plugins
2015-07-19 04:58:04.571 22991 WARNING neutron.api.extensions [-] Extension port-security not supported by any of loaded plugins
2015-07-19 04:58:04.583 22991 WARNING neutron.api.extensions [-] Extension router-service-type not supported by any of loaded plugins
2015-07-19 04:58:04.585 22991 WARNING neutron.api.extensions [-] Extension service-type not supported by any of loaded plugins
2015-07-19 04:58:04.586 22991 WARNING neutron.api.extensions [-] Extension vlan-transparent not supported by any of loaded plugins
2015-07-19 04:58:30.756 25811 WARNING neutron.agent.securitygroups_rpc [-] Driver configuration doesn't match with enable_security_group
2015-07-19 04:58:30.757 25811 WARNING neutron.agent.securitygroups_rpc [-] Driver configuration doesn't match with enable_security_group
2015-07-19 04:58:31.005 25811 WARNING neutron.agent.securitygroups_rpc [-] Driver configuration doesn't match with enable_security_group
2015-07-19 04:58:31.086 25811 WARNING neutron.api.extensions [-] Extension flavor not supported by any of loaded plugins
2015-07-19 04:58:31.093 25811 WARNING neutron.api.extensions [-] Extension metering not supported by any of loaded plugins
2015-07-19 04:58:31.095 25811 WARNING neutron.api.extensions [-] Extension port-security not supported by any of loaded plugins
2015-07-19 04:58:31.096 25811 WARNING neutron.api.extensions [-] Extension router-service-type not supported by any of loaded plugins
2015-07-19 04:58:31.098 25811 WARNING neutron.api.extensions [-] Extension service-type not supported by any of loaded plugins
2015-07-19 04:58:31.099 25811 WARNING neutron.api.extensions [-] Extension vlan-transparent not supported by any of loaded plugins
2015-07-19 04:59:01.809 25811 ERROR oslo_messaging.rpc.dispatcher [req-3c1bbaed-fd71-4962-8861-9b56c2189acc ] Exception during message handling: Agent with agent_type=L3 agent and host=overcloud-controller-1.localdomain could not be found
2015-07-19 04:59:01.809 25811 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-07-19 04:59:01.809 25811 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-07-19 04:59:01.809 25811 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
2015-07-19 04:59:01.809 25811 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-07-19 04:59:01.809 25811 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
2015-07-19 04:59:01.809 25811 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2015-07-19 04:59:01.809 25811 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2015-07-19 04:59:01.809 25811 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 81, in sync_routers
2015-07-19 04:59:01.809 25811 TRACE oslo_messaging.rpc.dispatcher     context, host, router_ids))
2015-07-19 04:59:01.809 25811 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/neutron/db/l3_agentschedulers_db.py", line 290, in list_active_sync_routers_on_active_l3_agent
2015-07-19 04:59:01.809 25811 TRACE oslo_messaging.rpc.dispatcher     context, constants.AGENT_TYPE_L3, host)
2015-07-19 04:59:01.809 25811 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/neutron/db/agents_db.py", line 197, in _get_agent_by_type_and_host
2015-07-19 04:59:01.809 25811 TRACE oslo_messaging.rpc.dispatcher     host=host)
2015-07-19 04:59:01.809 25811 TRACE oslo_messaging.rpc.dispatcher AgentNotFoundByTypeHost: Agent with agent_type=L3 agent and host=overcloud-controller-1.localdomain could not be found
2015-07-19 04:59:01.809 25811 TRACE oslo_messaging.rpc.dispatcher 
2015-07-19 04:59:01.828 25811 ERROR oslo_messaging._drivers.common [req-3c1bbaed-fd71-4962-8861-9b56c2189acc ] Returning exception Agent with agent_type=L3 agent and host=overcloud-controller-1.localdomain could not be found to caller
2015-07-19 04:59:01.828 25811 ERROR oslo_messaging._drivers.common [req-3c1bbaed-fd71-4962-8861-9b56c2189acc ] ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n    executor_callback))\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n    executor_callback)\n', '  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch\n    result = func(ctxt, **new_args)\n', '  File "/usr/lib/python2.7/site-packages/neutron/api/rpc/handlers/l3_rpc.py", line 81, in sync_routers\n    context, host, router_ids))\n', '  File "/usr/lib/python2.7/site-packages/neutron/db/l3_agentschedulers_db.py", line 290, in list_active_sync_routers_on_active_l3_agent\n    context, constants.AGENT_TYPE_L3, host)\n', '  File "/usr/lib/python2.7/site-packages/neutron/db/agents_db.py", line 197, in _get_agent_by_type_and_host\n    host=host)\n', 'AgentNotFoundByTypeHost: Agent with agent_type=L3 agent and host=overcloud-controller-1.localdomain could not be found\n']
2015-07-19 05:01:45.110 7727 WARNING neutron.agent.securitygroups_rpc [-] Driver configuration doesn't match with enable_security_group
2015-07-19 05:01:45.111 7727 WARNING neutron.agent.securitygroups_rpc [-] Driver configuration doesn't match with enable_security_group
2015-07-19 05:01:45.298 7727 WARNING neutron.agent.securitygroups_rpc [-] Driver configuration doesn't match with enable_security_group
2015-07-19 05:01:45.375 7727 WARNING neutron.api.extensions [-] Extension flavor not supported by any of loaded plugins
2015-07-19 05:01:45.382 7727 WARNING neutron.api.extensions [-] Extension metering not supported by any of loaded plugins
2015-07-19 05:01:45.384 7727 WARNING neutron.api.extensions [-] Extension port-security not supported by any of loaded plugins
2015-07-19 05:01:45.385 7727 WARNING neutron.api.extensions [-] Extension router-service-type not supported by any of loaded plugins
2015-07-19 05:01:45.386 7727 WARNING neutron.api.extensions [-] Extension service-type not supported by any of loaded plugins
2015-07-19 05:01:45.387 7727 WARNING neutron.api.extensions [-] Extension vlan-transparent not supported by any of loaded plugins
2015-07-19 05:03:59.742 18234 WARNING neutron.agent.securitygroups_rpc [-] Driver configuration doesn't match with enable_security_group
2015-07-19 05:03:59.742 18234 WARNING neutron.agent.securitygroups_rpc [-] Driver configuration doesn't match with enable_security_group
2015-07-19 05:03:59.942 18234 WARNING neutron.agent.securitygroups_rpc [-] Driver configuration doesn't match with enable_security_group
2015-07-19 05:04:00.021 18234 WARNING neutron.api.extensions [-] Extension flavor not supported by any of loaded plugins
2015-07-19 05:04:00.027 18234 WARNING neutron.api.extensions [-] Extension metering not supported by any of loaded plugins
2015-07-19 05:04:00.029 18234 WARNING neutron.api.extensions [-] Extension port-security not supported by any of loaded plugins
2015-07-19 05:04:00.030 18234 WARNING neutron.api.extensions [-] Extension router-service-type not supported by any of loaded plugins
2015-07-19 05:04:00.032 18234 WARNING neutron.api.extensions [-] Extension service-type not supported by any of loaded plugins
Comment 12 Ofer Blaut 2015-07-19 11:18:34 UTC 
Created attachment 1053604 [details]
logs of overcloud-controller-0


error are still seen in 
python-rdomanager-oscplugin-0.0.8-41.el7ost.noarch
Comment 13 Ofer Blaut 2015-07-19 11:19:39 UTC 
host=overcloud-controller-1.localdomain is not seen on overcloud-controller-0
Comment 14 Marios Andreou 2015-07-20 06:08:54 UTC 
Hi Ofer, not sure if this is still a problem - the error above may be because the l3 agent on controller-1 indeed is still coming up. At the end of the deploy can you please verify:

1. source overcloudrc; neutron agent-list
(you are looking for output like https://bugzilla.redhat.com/show_bug.cgi?id=1238117#c19 as an example... the agents should have host name like 'overcloud-controller-n.localdomain' and should all be :-)

2. pcs status | grep neutron -A 1
(on any of the controllers... they should all be started).

Let me know if there is still aproblem and I will try and reproduce,

thanks, marios
Comment 15 Udi Kalifon 2015-07-20 07:05:01 UTC 
Using python-rdomanager-oscplugin-0.0.8-32.el7ost.noarch:


[stack@puma01 ~]$ neutron agent-list
+---------+--------------------+------------------------------------+-------+--------+---------------------------+
| id      | agent_type         | host                               | alive | admin..| binary                    |
+---------+--------------------+------------------------------------+-------+--------+---------------------------+
| 1ae6... | Open vSwitch agent | neutron-n-0                        | :-)   | True   | neutron-openvswitch-agent |
| 1f12... | Metadata agent     | overcloud-controller-2.localdomain | :-)   | True   | neutron-metadata-agent    |
| 2af6... | Open vSwitch agent | overcloud-compute-0.localdomain    | :-)   | True   | neutron-openvswitch-agent |
| 392f... | Metadata agent     | overcloud-controller-0.localdomain | :-)   | True   | neutron-metadata-agent    |
| 40e6... | Open vSwitch agent | neutron-n-1                        | :-)   | True   | neutron-openvswitch-agent |
| 58b3... | L3 agent           | overcloud-controller-2.localdomain | :-)   | True   | neutron-l3-agent          |       
| 5e4a... | L3 agent           | neutron-n-1                        | :-)   | True   | neutron-l3-agent          |       
| 96a6... | Open vSwitch agent | neutron-n-2                        | :-)   | True   | neutron-openvswitch-agent |
| 9720... | Metadata agent     | neutron-n-1                        | :-)   | True   | neutron-metadata-agent    |
| a003... | L3 agent           | overcloud-controller-0.localdomain | :-)   | True   | neutron-l3-agent          |       
| adf4... | DHCP agent         | neutron-n-0                        | :-)   | True   | neutron-dhcp-agent        |       
| b967... | DHCP agent         | neutron-n-1                        | :-)   | True   | neutron-dhcp-agent        |       
| d94d... | L3 agent           | overcloud-controller-1.localdomain | xxx   | True   | neutron-l3-agent          |       
| eba1... | Metadata agent     | overcloud-controller-1.localdomain | xxx   | True   | neutron-metadata-agent    |
| ed49... | DHCP agent         | neutron-n-2                        | :-)   | True   | neutron-dhcp-agent        |       
+---------+--------------------+------------------------------------+-------+--------+---------------------------+


On controller 0:
================
[root@overcloud-controller-0 ~]# pcs status |grep neutron -A 1
 Clone Set: neutron-l3-agent-clone [neutron-l3-agent]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
--
 Clone Set: neutron-metadata-agent-clone [neutron-metadata-agent]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-ovs-cleanup-clone [neutron-ovs-cleanup]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-netns-cleanup-clone [neutron-netns-cleanup]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
--
 Clone Set: neutron-dhcp-agent-clone [neutron-dhcp-agent]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
--
 Clone Set: neutron-scale-clone [neutron-scale] (unique)
     neutron-scale:0	(ocf::neutron:NeutronScale):	Started overcloud-controller-0 
     neutron-scale:1	(ocf::neutron:NeutronScale):	Started overcloud-controller-1 
     neutron-scale:2	(ocf::neutron:NeutronScale):	Started overcloud-controller-2 
 Clone Set: neutron-openvswitch-agent-clone [neutron-openvswitch-agent]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
--
 Clone Set: neutron-server-clone [neutron-server]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
--
    neutron-openvswitch-agent_monitor_60000 on overcloud-controller-2 'OCF_PENDING' (196): call=231, status=complete, exit-reason='none', last-rc-change='Sun Jul 19 07:49:20 2015', queued=0ms, exec=0ms
    neutron-openvswitch-agent_monitor_60000 on overcloud-controller-1 'not running' (7): call=229, status=complete, exit-reason='none', last-rc-change='Sun Jul 19 07:50:04 2015', queued=0ms, exec=0ms
    neutron-openvswitch-agent_monitor_60000 on overcloud-controller-0 'OCF_PENDING' (196): call=230, status=complete, exit-reason='none', last-rc-change='Sun Jul 19 07:49:16 2015', queued=0ms, exec=0ms


On controller 1:
================
[root@overcloud-controller-1 ~]# pcs status |grep neutron -A 1
 Clone Set: neutron-l3-agent-clone [neutron-l3-agent]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
--
 Clone Set: neutron-metadata-agent-clone [neutron-metadata-agent]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-ovs-cleanup-clone [neutron-ovs-cleanup]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-netns-cleanup-clone [neutron-netns-cleanup]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]

--
 Clone Set: neutron-dhcp-agent-clone [neutron-dhcp-agent]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
--
 Clone Set: neutron-scale-clone [neutron-scale] (unique)
     neutron-scale:0	(ocf::neutron:NeutronScale):	Started overcloud-controller-0 
     neutron-scale:1	(ocf::neutron:NeutronScale):	Started overcloud-controller-1 
     neutron-scale:2	(ocf::neutron:NeutronScale):	Started overcloud-controller-2 
 Clone Set: neutron-openvswitch-agent-clone [neutron-openvswitch-agent]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
--
 Clone Set: neutron-server-clone [neutron-server]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
--
    neutron-openvswitch-agent_monitor_60000 on overcloud-controller-2 'OCF_PENDING' (196): call=231, status=complete, exit-reason='none', last-rc-change='Sun Jul 19 07:49:20 2015', queued=0ms, exec=0ms
    neutron-openvswitch-agent_monitor_60000 on overcloud-controller-1 'not running' (7): call=229, status=complete, exit-reason='none', last-rc-change='Sun Jul 19 07:50:04 2015', queued=0ms, exec=0ms
    neutron-openvswitch-agent_monitor_60000 on overcloud-controller-0 'OCF_PENDING' (196): call=230, status=complete, exit-reason='none', last-rc-change='Sun Jul 19 07:49:16 2015', queued=0ms, exec=0ms


On controller 2:
================
[root@overcloud-controller-2 ~]# pcs status |grep neutron -A 1
 Clone Set: neutron-l3-agent-clone [neutron-l3-agent]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
--
 Clone Set: neutron-metadata-agent-clone [neutron-metadata-agent]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-ovs-cleanup-clone [neutron-ovs-cleanup]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 Clone Set: neutron-netns-cleanup-clone [neutron-netns-cleanup]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
--
 Clone Set: neutron-dhcp-agent-clone [neutron-dhcp-agent]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
--
 Clone Set: neutron-scale-clone [neutron-scale] (unique)
     neutron-scale:0	(ocf::neutron:NeutronScale):	Started overcloud-controller-0 
     neutron-scale:1	(ocf::neutron:NeutronScale):	Started overcloud-controller-1 
     neutron-scale:2	(ocf::neutron:NeutronScale):	Started overcloud-controller-2 
 Clone Set: neutron-openvswitch-agent-clone [neutron-openvswitch-agent]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
--
 Clone Set: neutron-server-clone [neutron-server]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
--
    neutron-openvswitch-agent_monitor_60000 on overcloud-controller-2 'OCF_PENDING' (196): call=231, status=complete, exit-reason='none', last-rc-change='Sun Jul 19 07:49:20 2015', queued=0ms, exec=0ms
    neutron-openvswitch-agent_monitor_60000 on overcloud-controller-1 'not running' (7): call=229, status=complete, exit-reason='none', last-rc-change='Sun Jul 19 07:50:04 2015', queued=0ms, exec=0ms
    neutron-openvswitch-agent_monitor_60000 on overcloud-controller-0 'OCF_PENDING' (196): call=230, status=complete, exit-reason='none', last-rc-change='Sun Jul 19 07:49:16 2015', queued=0ms, exec=0ms
Comment 16 Marios Andreou 2015-07-20 07:11:51 UTC 
hey Udi, I think the 'fixed in ' info here is wrong (says python-rdomanager-oscplugin-0.0.8-21.el7ost at the top). The neutron agent list you have above tells me this environment still has neutronscale (agent hosts called 'neutron-n-' and also pcs 'neutron-scale')

The removal of neutronscale should be in openstack-tripleo-heat-templates-0.8.6-40.el7ost python-rdomanager-oscplugin-0.0.8-37.el7ost (see the related openstack-tripleo-heat-templates-0.8.6-40.el7ost python-rdomanager-oscplugin-0.0.8-37.el7ost )
Comment 17 Mike Burns 2015-07-20 11:31:26 UTC 
moving to ON_QA to retest with latest
Comment 18 Ofer Blaut 2015-07-20 13:36:35 UTC 
tested python-rdomanager-oscplugin-0.0.8-41.el7ost.noarch
[root@puma33 stack]# neutron agent-list
+--------------------------------------+--------------------+------------------------------------+-------+----------------+---------------------------+
| id                                   | agent_type         | host                               | alive | admin_state_up | binary                    |
+--------------------------------------+--------------------+------------------------------------+-------+----------------+---------------------------+
| 0ca6b48a-8d68-4ac7-8b31-28f326893979 | L3 agent           | overcloud-controller-1.localdomain | :-)   | True           | neutron-l3-agent          |
| 0dd19162-8710-45a1-beb6-9118a11875b8 | Open vSwitch agent | overcloud-controller-0.localdomain | :-)   | True           | neutron-openvswitch-agent |
| 0fdd2ba6-2748-4e6c-a3a3-dd452d16d253 | Open vSwitch agent | overcloud-controller-2.localdomain | :-)   | True           | neutron-openvswitch-agent |
| 194793a3-df95-4114-8f86-a78573e20ea3 | Open vSwitch agent | overcloud-compute-0.localdomain    | :-)   | True           | neutron-openvswitch-agent |
| 1fb5d8bc-ffe3-4e96-b174-d6d9747c7354 | Metadata agent     | overcloud-controller-0.localdomain | :-)   | True           | neutron-metadata-agent    |
| 219b56e8-7879-4573-b170-00c99156bf87 | DHCP agent         | overcloud-controller-2.localdomain | :-)   | True           | neutron-dhcp-agent        |
| 23156ca4-5fb2-439c-b677-ecf6bebb2f47 | Metadata agent     | overcloud-controller-2.localdomain | :-)   | True           | neutron-metadata-agent    |
| 2d7d76b0-5f8c-403d-b899-46fe8fbfffe9 | Metadata agent     | overcloud-controller-1.localdomain | :-)   | True           | neutron-metadata-agent    |
| 342b9bb2-1463-4607-8021-dad0a85444bf | L3 agent           | overcloud-controller-0.localdomain | :-)   | True           | neutron-l3-agent          |
| 3ea42cda-3e08-495d-85c1-683681cc1c8a | DHCP agent         | overcloud-controller-1.localdomain | :-)   | True           | neutron-dhcp-agent        |
| c36424ef-c406-49f4-bd94-b3598862d455 | L3 agent           | overcloud-controller-2.localdomain | :-)   | True           | neutron-l3-agent          |
| d9823577-02ba-4442-91a1-332888a9148c | Open vSwitch agent | overcloud-controller-1.localdomain | :-)   | True           | neutron-openvswitch-agent |
| df385119-2f7f-4098-83ea-8b30a795911a | DHCP agent         | overcloud-controller-0.localdomain | :-)   | True           | neutron-dhcp-agent        |
+--------------------------------------+--------------------+------------------------------------+-------+----------------+---------------------------+
Comment 20 errata-xmlrpc 2015-08-05 13:58:14 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2015:1549