Bug 1266910

Summary: All l3 agents go to standby ha_state after restarting haproxy resource
Product: Red Hat OpenStack Reporter: Marius Cornea <mcornea>
Component: rhosp-directorAssignee: Marios Andreou <mandreou>
Status: CLOSED ERRATA QA Contact: Marius Cornea <mcornea>
Severity: urgent Docs Contact:
Priority: medium    
Version: 7.0 (Kilo)CC: abeekhof, amuller, bperkins, dnavale, fdinitto, gfidente, hbrock, jguiditt, jhoffer, jslagle, majopela, mandreou, mangelajo, mburns, mcornea, michele, ohochman, rhel-osp-director-maint, rscarazz, vincent
Target Milestone: y2Keywords: Triaged
Target Release: 7.0 (Kilo)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-0.8.6-83.el7ost Doc Type: Bug Fix
Doc Text:
Previously, a Pacemaker constraint between the neutron-server and the neutron-ovs-cleanup processes meant that the over-cleanup restarted whenever the neutron-server did. The clearup by the ovs-cleanup (and associated netns-cleanup) should only occur when a node was being evacuated or rebooted and not for a neutron-server restart. As a result of this constraint, the neutron-server needed long startup to rebuild the data plane and ultimately caused issues for the dependent DHCP and L3 agents. With this update, the constraint between the neutron-server and the ovs-cleanup/netns-cleanup was removed. This means that the ovs-cleanup/net-ns cleanup does not re-run after neutron-server is restarted (for example, because haproxy was). The result is the two constraints chains for OpenStack Networking: ovs-cleanup-->netns-cleanup-->openvswitch-agent, and then neutron-server-->openvswitch-agent-->dhcp-agent-->l3-agent-->metadata-agent. As a result, when haproxy is restarted or when neutron-server is, more specifically, the ovs and netns cleanup scripts will not be re-run, so the OpenStack Networking service startup will proceed normally.
Story Points: ---
Clone Of:
: 1268244 (view as bug list) Environment:
Last Closed: 2015-12-21 16:49:25 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1262263, 1268244    
Attachments:
Description Flags
l3 agent logs
none
sos report controller0
none
sosreport controller1
none
sos report controller2 none

Description Marius Cornea 2015-09-28 13:27:25 UTC
Description of problem:
All the neutron l3 agents go to standby ha_state after restarting haproxy resource on one of the controllers. 

Version-Release number of selected component (if applicable):
openstack-heat-templates-0-0.6.20150605git.el7ost.noarch
openstack-tripleo-heat-templates-0.8.6-69.el7ost.noarch


How reproducible:
100%

Steps to Reproduce:
1. Deploy overcloud with 3 controllers 
2. Create tenant network, external network and router
3. Run pcs resource restart haproxy-clone on one of the controllers
4. Check l3 agents for the router 


Actual results:
All the 3 l3 agents show up as standby so connectivity with the router doesn't work.

Expected results:
One of the l3 agents is active. 

Additional info:
Before:
stack@instack:~>>> neutron l3-agent-list-hosting-router tenant-router
+--------------------------------------+------------------------------------+----------------+-------+----------+
| id                                   | host                               | admin_state_up | alive | ha_state |
+--------------------------------------+------------------------------------+----------------+-------+----------+
| 3cacef43-1cdd-496e-8dd0-ec6cf7541718 | overcloud-controller-1.localdomain | True           | :-)   | active   |
| 4d1a9ce9-0053-4695-a5e5-70f42005854e | overcloud-controller-0.localdomain | True           | :-)   | standby  |
| 03761081-95bb-4091-896f-3217421dc8fb | overcloud-controller-2.localdomain | True           | :-)   | standby  |
+--------------------------------------+------------------------------------+----------------+-------+----------+

[root@overcloud-controller-0 ~]# pcs resource restart haproxy-clone
haproxy-clone successfully restarted

stack@instack:~>>> neutron l3-agent-list-hosting-router tenant-router
+--------------------------------------+------------------------------------+----------------+-------+----------+
| id                                   | host                               | admin_state_up | alive | ha_state |
+--------------------------------------+------------------------------------+----------------+-------+----------+
| 3cacef43-1cdd-496e-8dd0-ec6cf7541718 | overcloud-controller-1.localdomain | True           | :-)   | standby  |
| 4d1a9ce9-0053-4695-a5e5-70f42005854e | overcloud-controller-0.localdomain | True           | :-)   | standby  |
| 03761081-95bb-4091-896f-3217421dc8fb | overcloud-controller-2.localdomain | True           | :-)   | standby  |
+--------------------------------------+------------------------------------+----------------+-------+----------+

Comment 2 Fabio Massimo Di Nitto 2015-09-28 14:22:38 UTC
Marius,

the action of: pcs restart haproxy-clone will temporary take down the whole openstack infrastructure and start it again. so yes, a temporary disconnection of the agents is to be expected.

The question is, what are you trying to achieve with this restart and why are you restarting in the first place?

Maybe the problem is simply procedural vs operational.

Comment 3 Fabio Massimo Di Nitto 2015-09-28 14:28:03 UTC
After talking on IRC with Mike,

please attach sosreports from all 3 controller nodes.

we need to see the current deployed configuration and any error in the logs.

Latest OSPd has been changing some pacemaker rules, and maybe there is a latent bug.

Comment 4 Miguel Angel Ajo 2015-09-28 14:38:54 UTC
Can you check if it's only a reporting thing, or if actually the low level keepalived's are not getting spawned, or none of them becoming master?

Comment 5 Marius Cornea 2015-09-28 15:13:23 UTC
It looks like none of the interfaces in the router namespace don't have assigned IP addresses except the ha-xx one. Also attaching the l3-agent logs

[root@overcloud-controller-0 ~]# ip netns exec qrouter-836c8dea-6d72-4346-bfaf-453c71cbdb1a ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
17: ha-2267c55d-c6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:de:f2:e8 brd ff:ff:ff:ff:ff:ff
    inet 169.254.192.3/18 brd 169.254.255.255 scope global ha-2267c55d-c6
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fede:f2e8/64 scope link 
       valid_lft forever preferred_lft forever
18: qr-411df5fd-83: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:c5:f7:cb brd ff:ff:ff:ff:ff:ff
    inet6 fe80::f816:3eff:fec5:f7cb/64 scope link 
       valid_lft forever preferred_lft forever
19: qg-5e15cd46-00: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:25:b2:79 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::f816:3eff:fe25:b279/64 scope link 
       valid_lft forever preferred_lft forever


[root@overcloud-controller-2 ~]# ip netns exec qrouter-836c8dea-6d72-4346-bfaf-453c71cbdb1a ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
17: ha-9d2e5780-f5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:a2:f5:7d brd ff:ff:ff:ff:ff:ff
    inet 169.254.192.2/18 brd 169.254.255.255 scope global ha-9d2e5780-f5
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fea2:f57d/64 scope link 
       valid_lft forever preferred_lft forever
18: qr-411df5fd-83: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:c5:f7:cb brd ff:ff:ff:ff:ff:ff
    inet6 fe80::f816:3eff:fec5:f7cb/64 scope link 
       valid_lft forever preferred_lft forever
19: qg-5e15cd46-00: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:25:b2:79 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::f816:3eff:fe25:b279/64 scope link 
       valid_lft forever preferred_lft forever

[root@overcloud-controller-1 ~]# ip netns exec qrouter-836c8dea-6d72-4346-bfaf-453c71cbdb1a ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
17: ha-05d65a7c-11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:2b:2a:4c brd ff:ff:ff:ff:ff:ff
    inet 169.254.192.1/18 brd 169.254.255.255 scope global ha-05d65a7c-11
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe2b:2a4c/64 scope link 
       valid_lft forever preferred_lft forever
18: qr-411df5fd-83: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:c5:f7:cb brd ff:ff:ff:ff:ff:ff
    inet6 fe80::f816:3eff:fec5:f7cb/64 scope link 
       valid_lft forever preferred_lft forever
19: qg-5e15cd46-00: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN 
    link/ether fa:16:3e:25:b2:79 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::f816:3eff:fe25:b279/64 scope link 
       valid_lft forever preferred_lft forever

Comment 6 Marius Cornea 2015-09-28 15:13:47 UTC
Created attachment 1077950 [details]
l3  agent logs

Comment 7 Marius Cornea 2015-09-28 15:24:00 UTC
Created attachment 1077952 [details]
sos report controller0

Comment 8 Marius Cornea 2015-09-28 15:25:52 UTC
Created attachment 1077954 [details]
sosreport controller1

Comment 9 Marius Cornea 2015-09-28 15:27:23 UTC
Created attachment 1077958 [details]
sos report controller2

Comment 10 Assaf Muller 2015-09-28 16:43:17 UTC
Maybe we can cut the process down and just give me an environment I can SSH in to?

Comment 12 Assaf Muller 2015-09-28 21:45:40 UTC
I played around with the system provided by Marius, which allowed me to cut down the needinfo-ping-pong by a couple of days to an hour.

Root cause analysis:
'pcs constraint order show' shows that restarting the haproxy resource (And zooming in specifically on Neutron resources) shows this chain of dependencies:

haproxy-clone -> openstack-keystone-clone -> neutron-server-clone -> neutron-ovs-cleanup-clone -> neutron-netns-cleanup-clone -> neutron-openvswitch-agent-clone -> neutron-dhcp-agent-clone -> neutron-l3-agent-clone -> neutron-metadata-agent-clone.

Cleaning that up to just the relevant parts:
neutron-ovs-cleanup-clone -> neutron-netns-cleanup-clone -> neutron-openvswitch-agent-clone -> neutron-l3-agent-clone.

Therefor this can be reproduced on devstack/packstack/RDO/OSP/whatever on juno, kilo, liberty or master as such:
1) Set up openstack from Juno+ on any distro with any installer with a minimum of two nodes, and have the L3 agent installed on both.
2) Create an HA router, observe everything is peachy.
3) Run 'neutron-ovs-cleanup', 'neutron-netns-cleanup', systemctl restart neutron-openvswitch-agent, systemctl restart neutron-l3-agent.
4) Observe the output of neutron l3-agent-list-hosting-router <router_id|router_name> (All standbys) and the 'ip a' output of all HA router namespaces (No IPv4 IPs on any interface in any namespace).

The issue is that ovs-cleanup deletes the router interfaces, which are then recreated by the OVS agent. The L3 agent is restarted, which SIGHUPS keepalived. On the backup nodes, addresses aren't assigned, which is fine. On the master node, it retains the master role, but doesn't configure the IP addresses since a state change hasn't occurred.

Solutions:
Keepalived is not at fault here, and arguably neither are the OVS or L3 Neutron agents. I don't understand why are the 'neutron-ovs-cleanup' and 'neutron-netns-cleanup' scripts managed by Pacemaker (And thus are invoked on a Neutron resource restart). If they weren't, this bug would not have existed. I think the reason they are is historical and relates to the way we used to do Neutron network node HA (With mucking around with the host names and whatnot), but that is no longer the case. I believe the solution is to unmanage these resources, delete their Pacemaker resources entirely, but make sure they're enabled in systemd so that they are executed when the node starts or is fenced.

Setting needinfo on Miguel to keep me honest on the proposed solution.

Comment 13 Andrew Beekhof 2015-09-29 01:30:44 UTC
keepalived and pacemaker are being used in the same deployment???

Comment 14 Assaf Muller 2015-09-29 03:04:55 UTC
keepalived is used by Neutron to float IPs between router namespaces, not for the OpenStack services themselves. Check out https://youtu.be/go4fOYOUkmE?t=7m52s.

Comment 15 Fabio Massimo Di Nitto 2015-09-29 04:20:24 UTC
(In reply to Assaf Muller from comment #12)
> I played around with the system provided by Marius, which allowed me to cut
> down the needinfo-ping-pong by a couple of days to an hour.
> 
> Root cause analysis:
> 'pcs constraint order show' shows that restarting the haproxy resource (And
> zooming in specifically on Neutron resources) shows this chain of
> dependencies:
> 
> haproxy-clone -> openstack-keystone-clone -> neutron-server-clone ->
> neutron-ovs-cleanup-clone -> neutron-netns-cleanup-clone ->
> neutron-openvswitch-agent-clone -> neutron-dhcp-agent-clone ->
> neutron-l3-agent-clone -> neutron-metadata-agent-clone.
> 
> Cleaning that up to just the relevant parts:
> neutron-ovs-cleanup-clone -> neutron-netns-cleanup-clone ->
> neutron-openvswitch-agent-clone -> neutron-l3-agent-clone.
> 
> Therefor this can be reproduced on devstack/packstack/RDO/OSP/whatever on
> juno, kilo, liberty or master as such:
> 1) Set up openstack from Juno+ on any distro with any installer with a
> minimum of two nodes, and have the L3 agent installed on both.
> 2) Create an HA router, observe everything is peachy.
> 3) Run 'neutron-ovs-cleanup', 'neutron-netns-cleanup', systemctl restart
> neutron-openvswitch-agent, systemctl restart neutron-l3-agent.
> 4) Observe the output of neutron l3-agent-list-hosting-router
> <router_id|router_name> (All standbys) and the 'ip a' output of all HA
> router namespaces (No IPv4 IPs on any interface in any namespace).
> 
> The issue is that ovs-cleanup deletes the router interfaces, which are then
> recreated by the OVS agent. The L3 agent is restarted, which SIGHUPS
> keepalived. On the backup nodes, addresses aren't assigned, which is fine.
> On the master node, it retains the master role, but doesn't configure the IP
> addresses since a state change hasn't occurred.
> 
> Solutions:
> Keepalived is not at fault here, and arguably neither are the OVS or L3
> Neutron agents. I don't understand why are the 'neutron-ovs-cleanup' and
> 'neutron-netns-cleanup' scripts managed by Pacemaker (And thus are invoked
> on a Neutron resource restart). If they weren't, this bug would not have
> existed. I think the reason they are is historical and relates to the way we
> used to do Neutron network node HA (With mucking around with the host names
> and whatnot), but that is no longer the case. I believe the solution is to
> unmanage these resources, delete their Pacemaker resources entirely, but
> make sure they're enabled in systemd so that they are executed when the node
> starts or is fenced.
> 
> Setting needinfo on Miguel to keep me honest on the proposed solution.

Even so, a reboot is no different than a restart.

If a reboot a machine or stop the cluster and start it again, the namespaces and ip addresses should be recreated either ways.

I have no issues to unplug to those 2 resources, but Miguel will have to confirm that it is the right thing to do, since this sequence come from you guys.

Comment 18 Miguel Angel Ajo 2015-09-29 13:22:47 UTC
The need of those cleanup scripts, is basically for the case when you move any serving agent (dhcp, l3) from host to host. 

They are just there to make sure that, if you move, for example the l3-agent from one host, to a new node (to resolve a node failure), the serving resources (namespaces and interfaces) will be cleaned (otherwise you have IP conflicts by router interfaces serving the same IPs, or DHCP services serving on the same IPs from different network nodes).

This was actually more important for Active/Passive deployments, which I'm not sure we support anymore for neutron in our installers.

In Active/Active I guess we could drop those, and just document it's a required step if you want to decommission a node which you're about to replace.

Comment 19 Miguel Angel Ajo 2015-09-29 13:40:55 UTC
On a side note, I believe the original chain of dependencies in the HA architecture was more like:

haproxy-clone -> openstack-keystone-clone -> neutron-server-clone  -> neutron-openvswitch-agent-clone -> neutron-dhcp-agent-clone -> neutron-l3-agent-clone -> neutron-metadata-agent-clone.

and:

openvswitch -> neutron-ovs-cleanup-clone -> neutron-netns-cleanup-clone -> neutron-openvswitch-agent-clone


But it seems the deployer is linearizing the dependencies.

Comment 23 Marios Andreou 2015-09-30 10:57:40 UTC
(In reply to Miguel Angel Ajo from comment #19)
> On a side note, I believe the original chain of dependencies in the HA
> architecture was more like:
> 
> haproxy-clone -> openstack-keystone-clone -> neutron-server-clone  ->
> neutron-openvswitch-agent-clone -> neutron-dhcp-agent-clone ->
> neutron-l3-agent-clone -> neutron-metadata-agent-clone.
> 
> and:
> 
> openvswitch -> neutron-ovs-cleanup-clone -> neutron-netns-cleanup-clone ->
> neutron-openvswitch-agent-clone
> 
> 
> But it seems the deployer is linearizing the dependencies.

Hi Miguel and Assaf thanks for the info and discussion here. At least with respect to [1] I believe the resource constraints are ok (I double checked, on our side, looks like [2] - at least the keystone->neutron-server start of chain)

In any case, and from the discussion here it seems desirable to remove the ovs and netns cleanup from that constraints chain - I am testing out what this looks like until the discussion winds down - hopefully can still remain part of the cluster, but as long as there isn't a dependency forcing netns and ovs-cleanup to also restart like when we restart neutron-server for example) - will have a review out as soon as I've tested things against latest poodle.

So to clarify, sounds to me like advice for now is to remove constraints involving those cleanup agents so that they aren't restarted when any of the other neutron agents or even server is.

thanks

[1] https://github.com/beekhof/osp-ha-deploy/blob/4d87a4a01be8fa787ed9ba19f6e91c79282668fc/pcmk/neutron-agents.scenario
[2] https://github.com/openstack/tripleo-heat-templates/blob/92fd515d2d544521b6c18e1ad63959c781c29672/puppet/manifests/overcloud_controller_pacemaker.pp#L1051

Comment 24 Fabio Massimo Di Nitto 2015-09-30 11:02:13 UTC
(In reply to marios from comment #23)
> (In reply to Miguel Angel Ajo from comment #19)
> > On a side note, I believe the original chain of dependencies in the HA
> > architecture was more like:
> > 
> > haproxy-clone -> openstack-keystone-clone -> neutron-server-clone  ->
> > neutron-openvswitch-agent-clone -> neutron-dhcp-agent-clone ->
> > neutron-l3-agent-clone -> neutron-metadata-agent-clone.
> > 
> > and:
> > 
> > openvswitch -> neutron-ovs-cleanup-clone -> neutron-netns-cleanup-clone ->
> > neutron-openvswitch-agent-clone
> > 
> > 
> > But it seems the deployer is linearizing the dependencies.
> 
> Hi Miguel and Assaf thanks for the info and discussion here. At least with
> respect to [1] I believe the resource constraints are ok (I double checked,
> on our side, looks like [2] - at least the keystone->neutron-server start of
> chain)
> 
> In any case, and from the discussion here it seems desirable to remove the
> ovs and netns cleanup from that constraints chain - I am testing out what
> this looks like until the discussion winds down - hopefully can still remain
> part of the cluster, but as long as there isn't a dependency forcing netns
> and ovs-cleanup to also restart like when we restart neutron-server for
> example) - will have a review out as soon as I've tested things against
> latest poodle.
> 
> So to clarify, sounds to me like advice for now is to remove constraints
> involving those cleanup agents so that they aren't restarted when any of the
> other neutron agents or even server is.


Please hold it. There has been some discussion on IRC related on how to change this in the correct way and Assaf is investigating some bits related to OSPd.

Dropping constraints can cause other issues due to parallel start of resources, that being systemd or pacemaker.

> 
> thanks
> 
> [1]
> https://github.com/beekhof/osp-ha-deploy/blob/
> 4d87a4a01be8fa787ed9ba19f6e91c79282668fc/pcmk/neutron-agents.scenario
> [2]
> https://github.com/openstack/tripleo-heat-templates/blob/
> 92fd515d2d544521b6c18e1ad63959c781c29672/puppet/manifests/
> overcloud_controller_pacemaker.pp#L1051

Comment 25 Marios Andreou 2015-09-30 11:06:01 UTC
(In reply to Fabio Massimo Di Nitto from comment #24)
> (In reply to marios from comment #23)
> > (In reply to Miguel Angel Ajo from comment #19)
> > > On a side note, I believe the original chain of dependencies in the HA
> > > architecture was more like:
> > > 
> > > haproxy-clone -> openstack-keystone-clone -> neutron-server-clone  ->
> > > neutron-openvswitch-agent-clone -> neutron-dhcp-agent-clone ->
> > > neutron-l3-agent-clone -> neutron-metadata-agent-clone.
> > > 
> > > and:
> > > 
> > > openvswitch -> neutron-ovs-cleanup-clone -> neutron-netns-cleanup-clone ->
> > > neutron-openvswitch-agent-clone
> > > 
> > > 
> > > But it seems the deployer is linearizing the dependencies.
> > 
> > Hi Miguel and Assaf thanks for the info and discussion here. At least with
> > respect to [1] I believe the resource constraints are ok (I double checked,
> > on our side, looks like [2] - at least the keystone->neutron-server start of
> > chain)
> > 
> > In any case, and from the discussion here it seems desirable to remove the
> > ovs and netns cleanup from that constraints chain - I am testing out what
> > this looks like until the discussion winds down - hopefully can still remain
> > part of the cluster, but as long as there isn't a dependency forcing netns
> > and ovs-cleanup to also restart like when we restart neutron-server for
> > example) - will have a review out as soon as I've tested things against
> > latest poodle.
> > 
> > So to clarify, sounds to me like advice for now is to remove constraints
> > involving those cleanup agents so that they aren't restarted when any of the
> > other neutron agents or even server is.
> 
> 
> Please hold it. There has been some discussion on IRC related on how to
> change this in the correct way and Assaf is investigating some bits related
> to OSPd.
> 
> Dropping constraints can cause other issues due to parallel start of
> resources, that being systemd or pacemaker.

Yes of course Fabio, I didn't mean to imply I was pressing ahead or ignoring your comments, you make excellent counter points... I am just preparing in case we need it NOW() ... I am just testing out what this looks like until the discussion winds down

It isn't even posted upstream (currently http://paste.fedoraproject.org/273099/14436101/raw/ for example), and even if it were upstream you could just veto it there :)

Comment 28 Marios Andreou 2015-09-30 15:03:12 UTC
(In reply to Fabio Massimo Di Nitto from comment #24)
> (In reply to marios from comment #23)
> > (In reply to Miguel Angel Ajo from comment #19)
> > > On a side note, I believe the original chain of dependencies in the HA
> > > architecture was more like:
> > > 
> > > haproxy-clone -> openstack-keystone-clone -> neutron-server-clone  ->
> > > neutron-openvswitch-agent-clone -> neutron-dhcp-agent-clone ->
> > > neutron-l3-agent-clone -> neutron-metadata-agent-clone.
> > > 
> > > and:
> > > 
> > > openvswitch -> neutron-ovs-cleanup-clone -> neutron-netns-cleanup-clone ->
> > > neutron-openvswitch-agent-clone
> > > 
> > > 
> > > But it seems the deployer is linearizing the dependencies.
> > 
> > Hi Miguel and Assaf thanks for the info and discussion here. At least with
> > respect to [1] I believe the resource constraints are ok (I double checked,
> > on our side, looks like [2] - at least the keystone->neutron-server start of
> > chain)
> > 
> > In any case, and from the discussion here it seems desirable to remove the
> > ovs and netns cleanup from that constraints chain - I am testing out what
> > this looks like until the discussion winds down - hopefully can still remain
> > part of the cluster, but as long as there isn't a dependency forcing netns
> > and ovs-cleanup to also restart like when we restart neutron-server for
> > example) - will have a review out as soon as I've tested things against
> > latest poodle.
> > 
> > So to clarify, sounds to me like advice for now is to remove constraints
> > involving those cleanup agents so that they aren't restarted when any of the
> > other neutron agents or even server is.
> 
> 
> Please hold it. There has been some discussion on IRC related on how to
> change this in the correct way and Assaf is investigating some bits related
> to OSPd.
> 
> Dropping constraints can cause other issues due to parallel start of
> resources, that being systemd or pacemaker.

JUST for reference and so we can have something to point at (whether as a good or a bad example ;) ) I have posted a review upstream at https://review.openstack.org/#/c/229466/ (and filed upstream bug at https://bugs.launchpad.net/tripleo/+bug/1501378 ).

FTR, I tested this applied to latest poodle tripleo heat templates... when you restart neutron-server the ovs/netns cleanup do not get restarted, same when you do a full haproxy restart. I also didn't lose my tenant router and the IP continued to be hosted on controller-0 without problems.

thanks

Comment 30 Miguel Angel Ajo 2015-10-01 15:11:52 UTC
(In reply to marios from comment #28)
> https://review.openstack.org/#/c/229466/ (and filed upstream bug at
> https://bugs.launchpad.net/tripleo/+bug/1501378 ).
> 
> FTR, I tested this applied to latest poodle tripleo heat templates... when
> you restart neutron-server the ovs/netns cleanup do not get restarted, same
> when you do a full haproxy restart. I also didn't lose my tenant router and
> the IP continued to be hosted on controller-0 without problems.
> 
> thanks


Thanks Marios!,

   That's the right order of dependencies, the mistake was also in the OSP6 HA architecture, which *I* reviewed. I wonder how did I miss this little detail.


   How does it sound to fix this order in the installer, and also provide a note to customers to delete the wrong order constraints on OSP6 or OSP7 (pre-this-patch) installations?.

   We could work-around with the pcmk resource agents for (netns/ovs cleanup) but I'd feel more confortable with the right order of dependencies.


Best,
Miguel Ángel.

Comment 31 Raoul Scarazzini 2015-10-02 09:32:10 UTC
After a call with Miguel, Michele and Fabio we ended up to this solution.
At the moment this is the order sequence of all the interested components:

haproxy-clone
 keystone-clone
  neutron-server-clone
   neutron-ovs-cleanup-clone
    neutron-netns-cleanup-clone
     neutron-openvswitch-agent-clone
      neutron-dhcp-agent-clone      
       neutron-l3-agent-clone
        neutron-metadata-agent-clone

This is the desiderata:

neutron-ovs-cleanup-clone
 neutron-netns-cleanup-clone
  neutron-openvswitch-agent-clone
haproxy-clone
 keystone-clone
  neutron-server-clone
   neutron-openvswitch-agent-clone
    neutron-dhcp-agent-clone      
     neutron-l3-agent-clone
      neutron-metadata-agent-clone

So the sequence of requested actions will be:

1) Remove the neutron-ovs-cleanup-clone and neutron-netns-cleanup-clone dependency from neutron-server-clone;
2) Add a neutron-openvswitch-agent-clone dependency from neutron-server-clone;

In this way the cleanup actions will be started and stopped when the cluster comes up or goes down.

And that's all. There are also a set optimizations that are not critical and will be done later (they are just logical). We will open an RFE for that.

Comment 32 Andrew Beekhof 2015-10-05 00:31:03 UTC
Is there any way for pcs resource cleanup to trigger a restart of neutron-*-cleanup ?
If so then you're still going to hit the problem described in comment #12

> 3) Run 'neutron-ovs-cleanup', 'neutron-netns-cleanup', systemctl restart neutron-openvswitch-agent, systemctl restart neutron-l3-agent.
> 4) Observe the output of neutron l3-agent-list-hosting-router <router_id|router_name> (All standbys) and the 'ip a' output of all HA router namespaces (No IPv4 IPs on any interface in any namespace).

Comment 33 Marios Andreou 2015-10-05 13:51:10 UTC
(In reply to Raoul Scarazzini from comment #31)
> After a call with Miguel, Michele and Fabio we ended up to this solution.
> At the moment this is the order sequence of all the interested components:
> 
> haproxy-clone
>  keystone-clone
>   neutron-server-clone
>    neutron-ovs-cleanup-clone
>     neutron-netns-cleanup-clone
>      neutron-openvswitch-agent-clone
>       neutron-dhcp-agent-clone      
>        neutron-l3-agent-clone
>         neutron-metadata-agent-clone
> 
> This is the desiderata:
> 
> neutron-ovs-cleanup-clone
>  neutron-netns-cleanup-clone
>   neutron-openvswitch-agent-clone
> haproxy-clone
>  keystone-clone
>   neutron-server-clone
>    neutron-openvswitch-agent-clone
>     neutron-dhcp-agent-clone      
>      neutron-l3-agent-clone
>       neutron-metadata-agent-clone


afaics this ^^^ is done in  https://review.openstack.org/#/c/229466/ does that look ok - i will take out of WIP for review momentarily

Comment 34 Raoul Scarazzini 2015-10-12 16:20:44 UTC
@marios: testing the patch on a brand new installation give us the desiderata, so once the patch will go on package we will have what we want.

@Andrew: for what we see the relations now are fine and the only way of breaking something is to manually launch a cleanup (or a start, or a stop) on the cleanup resources. Are we missing something?

Comment 35 Andrew Beekhof 2015-10-12 23:25:12 UTC
(In reply to Raoul Scarazzini from comment #34)
> @marios: testing the patch on a brand new installation give us the
> desiderata, so once the patch will go on package we will have what we want.
> 
> @Andrew: for what we see the relations now are fine and the only way of
> breaking something is to manually launch a cleanup (or a start, or a stop)
> on the cleanup resources. Are we missing something?

From what I've seen, admins run 'pcs resource cleanup' /all the time/.

So if that will cause the neutron-*-cleanup-clone resource to be started again - you still have a big problem.  Make sure the agents create a lockfile so that their non-recurring monitor actions can return OCF_RUNNING when appropriate.

BUT also make sure that its in a location that gets erased when the node reboots.

Comment 36 Miguel Angel Ajo 2015-10-14 08:02:47 UTC
(In reply to Andrew Beekhof from comment #35)
> (In reply to Raoul Scarazzini from comment #34)
> > @marios: testing the patch on a brand new installation give us the
> > desiderata, so once the patch will go on package we will have what we want.
> > 
> > @Andrew: for what we see the relations now are fine and the only way of
> > breaking something is to manually launch a cleanup (or a start, or a stop)
> > on the cleanup resources. Are we missing something?
> 
> From what I've seen, admins run 'pcs resource cleanup' /all the time/.

Isn't that supposed to just check all services status and cleanup the history?, that should be ok with netns-cleanup [1], and ovs-cleanup [2]

> 
> So if that will cause the neutron-*-cleanup-clone resource to be started
> again - you still have a big problem.  Make sure the agents create a
> lockfile so that their non-recurring monitor actions can return OCF_RUNNING
> when appropriate.

I think that's the case

> 
> BUT also make sure that its in a location that gets erased when the node
> reboots.

I think we're clear, but please review [1] [2] to be safe.

[1] https://github.com/openstack-packages/neutron/blob/rpm-master/neutron-netns-cleanup.init#L32 (wrapped by: https://github.com/openstack-packages/neutron/blob/rpm-master/NetnsCleanup.ocf_ra) 

[2] https://github.com/openstack-packages/neutron/blob/rpm-master/neutron-ovs-cleanup.init#L33  (wrapped by: https://github.com/openstack-packages/neutron/blob/rpm-master/OVSCleanup.ocf_ra ) 


eventually we could get rid of those, by relying on systemd, if we wanted, but the previous call of the HA team, was not to mess up to avoid breaking upgrade paths. IMO we could eventually switch it.

https://github.com/openstack-packages/neutron/blob/rpm-master/neutron-netns-cleanup.service

and 

https://github.com/openstack-packages/neutron/blob/rpm-master/neutron-ovs-cleanup.service

oneshot & RemainAfterExit services.

Comment 37 Fabio Massimo Di Nitto 2015-10-14 08:16:36 UTC
(In reply to Miguel Angel Ajo from comment #36)
> (In reply to Andrew Beekhof from comment #35)
> > (In reply to Raoul Scarazzini from comment #34)
> > > @marios: testing the patch on a brand new installation give us the
> > > desiderata, so once the patch will go on package we will have what we want.
> > > 
> > > @Andrew: for what we see the relations now are fine and the only way of
> > > breaking something is to manually launch a cleanup (or a start, or a stop)
> > > on the cleanup resources. Are we missing something?
> > 
> > From what I've seen, admins run 'pcs resource cleanup' /all the time/.
> 
> Isn't that supposed to just check all services status and cleanup the
> history?, that should be ok with netns-cleanup [1], and ovs-cleanup [2]
> 
> > 
> > So if that will cause the neutron-*-cleanup-clone resource to be started
> > again - you still have a big problem.  Make sure the agents create a
> > lockfile so that their non-recurring monitor actions can return OCF_RUNNING
> > when appropriate.
> 
> I think that's the case
> 
> > 
> > BUT also make sure that its in a location that gets erased when the node
> > reboots.
> 
> I think we're clear, but please review [1] [2] to be safe.
> 
> [1]
> https://github.com/openstack-packages/neutron/blob/rpm-master/neutron-netns-
> cleanup.init#L32 (wrapped by:
> https://github.com/openstack-packages/neutron/blob/rpm-master/NetnsCleanup.
> ocf_ra) 
> 
> [2]
> https://github.com/openstack-packages/neutron/blob/rpm-master/neutron-ovs-
> cleanup.init#L33  (wrapped by:
> https://github.com/openstack-packages/neutron/blob/rpm-master/OVSCleanup.
> ocf_ra ) 
> 
> 
> eventually we could get rid of those, by relying on systemd, if we wanted,
> but the previous call of the HA team, was not to mess up to avoid breaking
> upgrade paths. IMO we could eventually switch it.
> 
> https://github.com/openstack-packages/neutron/blob/rpm-master/neutron-netns-
> cleanup.service
> 
> and 
> 
> https://github.com/openstack-packages/neutron/blob/rpm-master/neutron-ovs-
> cleanup.service
> 
> oneshot & RemainAfterExit services.

As I already explained, we cannot switch to systemd services outside of cluster management. Even using the unit files won´t help because it´s racy.

Comment 38 Andrew Beekhof 2015-10-14 09:53:38 UTC
(In reply to Miguel Angel Ajo from comment #36)
> (In reply to Andrew Beekhof from comment #35)
> > (In reply to Raoul Scarazzini from comment #34)
> > > @marios: testing the patch on a brand new installation give us the
> > > desiderata, so once the patch will go on package we will have what we want.
> > > 
> > > @Andrew: for what we see the relations now are fine and the only way of
> > > breaking something is to manually launch a cleanup (or a start, or a stop)
> > > on the cleanup resources. Are we missing something?
> > 
> > From what I've seen, admins run 'pcs resource cleanup' /all the time/.
> 
> Isn't that supposed to just check all services status and cleanup the
> history?, that should be ok with netns-cleanup [1], and ovs-cleanup [2]

It erases the clusters knowledge of the resource's state.
The cluster will then ask the resource what state it is in.

If the agent says "Stopped", we're going to start it again which I understand would cause problems.

Hence why I'm highlighting the need to know, independent of the cluster, what state you are in.

If you've got that covered, then great.
Unfortunately many people forget this part so I don't like to assume :) 

> 
> > 
> > So if that will cause the neutron-*-cleanup-clone resource to be started
> > again - you still have a big problem.  Make sure the agents create a
> > lockfile so that their non-recurring monitor actions can return OCF_RUNNING
> > when appropriate.
> 
> I think that's the case
> 
> > 
> > BUT also make sure that its in a location that gets erased when the node
> > reboots.
> 
> I think we're clear, but please review [1] [2] to be safe.
> 
> [1]
> https://github.com/openstack-packages/neutron/blob/rpm-master/neutron-netns-
> cleanup.init#L32 (wrapped by:
> https://github.com/openstack-packages/neutron/blob/rpm-master/NetnsCleanup.
> ocf_ra) 
> 
> [2]
> https://github.com/openstack-packages/neutron/blob/rpm-master/neutron-ovs-
> cleanup.init#L33  (wrapped by:
> https://github.com/openstack-packages/neutron/blob/rpm-master/OVSCleanup.
> ocf_ra ) 

by eye they look ok

Comment 40 James Slagle 2015-11-20 18:01:09 UTC
Moving back to ASSIGNED as I think this fix is incomplete.

We need to also script the changes to the neutron resource constraints in yum_update.sh so that the right contraints are in place prior to doing a yum update.

We did this for the other set of constraints that was missing from 7.0 in:
https://review.openstack.org/#/c/244321
(and the corresponding downstream patch https://code.engineering.redhat.com/gerrit/#/c/61871/)

I think we need to do the same for these as well.

Comment 42 Marius Cornea 2015-12-09 20:21:38 UTC
stack@instack:~>>> rpm -qa | grep openstack-tripleo-heat-templates
openstack-tripleo-heat-templates-0.8.6-87.el7ost.noarch
stack@instack:~>>> 
stack@instack:~>>> neutron l3-agent-list-hosting-router tenant-router
+--------------------------------------+------------------------------------+----------------+-------+----------+
| id                                   | host                               | admin_state_up | alive | ha_state |
+--------------------------------------+------------------------------------+----------------+-------+----------+
| 741dfed3-4ece-4c99-9d59-0c6e0f2d25fd | overcloud-controller-2.localdomain | True           | :-)   | standby  |
| 41daa68c-e287-43dd-9ce3-4ad10fdd297e | overcloud-controller-0.localdomain | True           | :-)   | standby  |
| be167a58-0dff-422e-b54d-91f8dcc055d9 | overcloud-controller-1.localdomain | True           | :-)   | active   |
+--------------------------------------+------------------------------------+----------------+-------+----------+

[root@overcloud-controller-0 ~]# pcs resource restart haproxy-clone

stack@instack:~>>> neutron l3-agent-list-hosting-router tenant-router
+--------------------------------------+------------------------------------+----------------+-------+----------+
| id                                   | host                               | admin_state_up | alive | ha_state |
+--------------------------------------+------------------------------------+----------------+-------+----------+
| 741dfed3-4ece-4c99-9d59-0c6e0f2d25fd | overcloud-controller-2.localdomain | True           | :-)   | standby  |
| 41daa68c-e287-43dd-9ce3-4ad10fdd297e | overcloud-controller-0.localdomain | True           | :-)   | active   |
| be167a58-0dff-422e-b54d-91f8dcc055d9 | overcloud-controller-1.localdomain | True           | :-)   | standby  |
+--------------------------------------+------------------------------------+----------------+-------+----------+

Comment 44 errata-xmlrpc 2015-12-21 16:49:25 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2015:2650