Bug 1036523

Would be good to understand a couple of things here - (a) whether this is specific to Neutron (maybe it has an older copy of the RPC code, or maybe this is an issue for all services) and (b) whether the RabbitMQ support is any better

(In reply to Mark McLoughlin from comment #1)
> Would be good to understand a couple of things here - (a) whether this is
> specific to Neutron (maybe it has an older copy of the RPC code, or maybe
> this is an issue for all services) and (b) whether the RabbitMQ support is
> any better


http://rhel-ha.etherpad.corp.redhat.com/RHOS-RHEL-HA-how-to

and https://bugzilla.redhat.com/show_bug.cgi?id=1036518

have all the info to reproduce.

You will need to unleash somebody to test here != me :)

(In reply to Mark McLoughlin from comment #1)
> Would be good to understand a couple of things here - (a) whether this is
> specific to Neutron (maybe it has an older copy of the RPC code, or maybe
> this is an issue for all services) and (b) whether the RabbitMQ support is
> any better

a) All services are affected.

I've reviewed neutron's copy of openstack.common, the oslo-incubator repo, and the oslo.messaging repo. In all cases, both the qpid and rabbit drivers support only a single concurrent connection.  If provided, multiple hosts will be used to perform round-robin connection attempts (either at startup or on connection failure) until a successful connection is made.


b) RabbitMQ isn't better

- Both drivers immediately attempt to reconnect if their connection to the messaging service drops due to service failure.  
- Both drivers use an increasing reconnection interval to limit the number of client connection attempts in the event of a service disruption.
- Both drivers will detect communication failure on send and attempt to reconnect.

Only the qpid driver supports a connection liveness check.  The interval is configured via qpid_heartbeat (default=60).  The ~2m that it was taking for to failover was likely related to this interval.  Care is advised in reducing this interval, though, since it will likely increase the idle load on the qpidd service.

Mark: Can you please advise as to whether it should be considered a bug that openstack services are not capable of maintaining multiple connections to an AMQP service?  I am thinking it should be the responsibility of an HA implementation rather than something builtin to oslo.

In my testing, the Neutron service immediately attempts reconnection if the connection to qpid is broken by stopping qpidd.  Can you please verify this result (which would render symptom #2 invalid)?  I suspect that the use of iptables filtering was not accurately simulating service failure and leaving some open connections, requiring a heartbeat failure (default interval -> 60s) to discover the service disruption.

(In reply to Maru Newby from comment #4)
> Mark: Can you please advise as to whether it should be considered a bug that
> openstack services are not capable of maintaining multiple connections to an
> AMQP service?  I am thinking it should be the responsibility of an HA
> implementation rather than something builtin to oslo.

https://bugzilla.redhat.com/show_bug.cgi?id=1036518

this was indeed the original idea, but it appears that using haproxy to qpid instances is also broken.

(In reply to Maru Newby from comment #5)
> In my testing, the Neutron service immediately attempts reconnection if the
> connection to qpid is broken by stopping qpidd.

stopping qpid will close the TCP connection properly instead of triggering a timeout.

>  Can you please verify this
> result (which would render symptom #2 invalid)?  I suspect that the use of
> iptables filtering was not accurately simulating service failure and leaving
> some open connections, requiring a heartbeat failure (default interval ->
> 60s) to discover the service disruption.

Nope, you are comparing oranges and apples :)

You can´t predict that the qpid service will always shutdown properly and terminate the TCP connections correctly.

A network disconnection will trigger a timeout similar to iptables (try for example unplugging the network cable, this can be simulated also in virt environment by manually disconnecting the vm virt interface from the configured bridge)

I am 100% sure that my iptables filtering are correct. I use those same rules to perform lots of other tests.

qpid is on 192.168.1.1

neutron on 192.168.1.2

on qpid host make sure there are no iptables rules installed:

for rule in INPUT FORWARD OUTPUT; do
 iptables -F $rule
done

iptables -A INPUT -j DROP -s 192.168.1.2
iptables -A OUTPUT -j DROP -d 192.168.1.2

This reminds me of this rabbitmq driver bug - https://bugs.launchpad.net/oslo/+bug/856764

I've lost track of the details of the debate, but the requirement is simple - you need some form of heatbeat or keepalive so that you quickly notice the connection has stalled and that you need to failover

If qpid_heartbeat=60 and we're taking 2 minutes to fail over, that suggests we're attempting to reconnect to the same host as our failed connection - it could be an easy fix to cut the failover time in half

And if 60 seconds is too slow a fail over time, we should reduce qpid_heatbeat to whatever is acceptable. There's no magic here - you can only know that the other side has gone away if it fails to respond to you.

Oh, please do open upstream oslo.messaging bugs for this as appropriate

Hope that helps ...

lpeer mentioned over email that this might be an HAProxy bug.  The comments on this bug don't convince me that this is a HA Proxy bug... 

It should be possible for a service to have a list of qpid brokers, perform liveness checks and failover in a round robin fashion.  At least, this is my understanding from multiple conversations with the qpid folks.  Requiring a load balancer in between to perform this round robin failover seems a bit heavyweight.

If the qpid driver in Oslo (copied to all of the services) can't properly handle broker failover this way, then I think that needs to be fixed.

Would like some input here from folks on the qpid team and/or folks familiar with the qpid drivers in OpenStack.

As a completely separate issues, as Comment #6 indicates, we were not able to get HAProxy to load balance qpid connections to work around the fact that the qpid broker round robin mechanism in the qpid driver itself was not working.  I think these two bugs are separate, and I don't think having to put HAProxy in between each OpenStack qpid client and qpid broker should be necessary, unless the qpid experts tell me that this is the only way to provide round-robin failover for qpid.

(In reply to Perry Myers from comment #10)
> lpeer mentioned over email that this might be an HAProxy bug.  The comments
> on this bug don't convince me that this is a HA Proxy bug... 

Neither bugs are haproxy really.

The bug mentioned in comment #6 is specific in combining the qpidd driver used by nova/neutron with haproxy to qpid

This is specific to the direct connection between neutron/nova to qpidd instances.

We have to make sure NOT to mix the two issues here (in fact, you mentioned the same below.. :))

> 
> It should be possible for a service to have a list of qpid brokers, perform
> liveness checks and failover in a round robin fashion.  At least, this is my
> understanding from multiple conversations with the qpid folks.  Requiring a
> load balancer in between to perform this round robin failover seems a bit
> heavyweight.

In fact, it should not be required. The problem with the current implementation, from a HA perspective, is the time it takes for the qpid driver in Oslo to recognize a server has gone away and reconnect to the next, and from a LB perspective, the fact that the connections are not load-balanced as expected.

> 
> If the qpid driver in Oslo (copied to all of the services) can't properly
> handle broker failover this way, then I think that needs to be fixed.
> 
> Would like some input here from folks on the qpid team and/or folks familiar
> with the qpid drivers in OpenStack.
> 
> As a completely separate issues, as Comment #6 indicates, we were not able
> to get HAProxy to load balance qpid connections to work around the fact that
> the qpid broker round robin mechanism in the qpid driver itself was not
> working.  I think these two bugs are separate, and I don't think having to
> put HAProxy in between each OpenStack qpid client and qpid broker should be
> necessary, unless the qpid experts tell me that this is the only way to
> provide round-robin failover for qpid.

Ok we agree. they are 2 separate issues.

The use of haproxy in between, makes it easier to failover and configure (and reconfigure) most of the services. Since we need to setup haproxy for another bug in glance <-> qpid, re-using the same channel for other services simplifies the overall configuration steps and keep them consistent across all systems/subsystems.

(In reply to Mark McLoughlin from comment #8)
> This reminds me of this rabbitmq driver bug -
> https://bugs.launchpad.net/oslo/+bug/856764
> 
> I've lost track of the details of the debate, but the requirement is simple
> - you need some form of heatbeat or keepalive so that you quickly notice the
> connection has stalled and that you need to failover
> 
> If qpid_heartbeat=60 and we're taking 2 minutes to fail over, that suggests
> we're attempting to reconnect to the same host as our failed connection - it
> could be an easy fix to cut the failover time in half

The way heartbeat is defined for AMQP 0-10 is that the connection should be considered failed if *two* heartbeat intervals pass without any  heartbeats. SO if you want to detect a failure within 60 seconds, the heartbeat interval should be 30secs.

(Note also that the heartbeats are only sent if there is no other traffic on the connection).

(In reply to Fabio Massimo Di Nitto from comment #11)
> (In reply to Perry Myers from comment #10)
> > It should be possible for a service to have a list of qpid brokers, perform
> > liveness checks and failover in a round robin fashion.  At least, this is my
> > understanding from multiple conversations with the qpid folks.  Requiring a
> > load balancer in between to perform this round robin failover seems a bit
> > heavyweight.
> 
> In fact, it should not be required. The problem with the current
> implementation, from a HA perspective, is the time it takes for the qpid
> driver in Oslo to recognize a server has gone away and reconnect to the
> next, and from a LB perspective, the fact that the connections are not
> load-balanced as expected.

The time to detect failover is controlled using the heartbeat interval, with two intervals passing before the connection is treated as failed.

In terms of load-balancing, that would only work if a clustered qpidd is used. In that case, though load balancing will have some impact, the overall throughput is generally noticeably reduced as compared to a single standalone instance anyway. If instead pacemaker is used to start a backup unclustered qpidd, then load balancing isn't relevant.

(In reply to Gordon Sim from comment #14)
> (In reply to Fabio Massimo Di Nitto from comment #11)
> > (In reply to Perry Myers from comment #10)
> > > It should be possible for a service to have a list of qpid brokers, perform
> > > liveness checks and failover in a round robin fashion.  At least, this is my
> > > understanding from multiple conversations with the qpid folks.  Requiring a
> > > load balancer in between to perform this round robin failover seems a bit
> > > heavyweight.
> > 
> > In fact, it should not be required. The problem with the current
> > implementation, from a HA perspective, is the time it takes for the qpid
> > driver in Oslo to recognize a server has gone away and reconnect to the
> > next, and from a LB perspective, the fact that the connections are not
> > load-balanced as expected.
> 
> The time to detect failover is controlled using the heartbeat interval, with
> two intervals passing before the connection is treated as failed.
> 
> In terms of load-balancing, that would only work if a clustered qpidd is
> used.

It is the case.

> In that case, though load balancing will have some impact, the overall
> throughput is generally noticeably reduced as compared to a single
> standalone instance anyway.

right now i am more worried about basic working vs performances.

> If instead pacemaker is used to start a backup
> unclustered qpidd, then load balancing isn't relevant.

Yes, I am filing those bugs as I find issues deploying RHOS on top of RHEL-HA/LB.

(In reply to Mark McLoughlin from comment #8)
> This reminds me of this rabbitmq driver bug -
> https://bugs.launchpad.net/oslo/+bug/856764
> 
> I've lost track of the details of the debate, but the requirement is simple
> - you need some form of heatbeat or keepalive so that you quickly notice the
> connection has stalled and that you need to failover
> 
> If qpid_heartbeat=60 and we're taking 2 minutes to fail over, that suggests
> we're attempting to reconnect to the same host as our failed connection - it
> could be an easy fix to cut the failover time in half
> 
> And if 60 seconds is too slow a fail over time, we should reduce
> qpid_heatbeat to whatever is acceptable. There's no magic here - you can
> only know that the other side has gone away if it fails to respond to you.
> 
> Oh, please do open upstream oslo.messaging bugs for this as appropriate
> 
> Hope that helps ...

Your assumptions as to the failover time were correct. Both the rabbit and qpid drivers reconnect in the order in which the AMQP servers are defined rather than connecting to the next server that did not fail.
 
I've filed a launchpad bug as requested and added a reference to this bz.

(In reply to Gordon Sim from comment #13)
> (In reply to Mark McLoughlin from comment #8)
> > This reminds me of this rabbitmq driver bug -
> > https://bugs.launchpad.net/oslo/+bug/856764
> > 
> > I've lost track of the details of the debate, but the requirement is simple
> > - you need some form of heatbeat or keepalive so that you quickly notice the
> > connection has stalled and that you need to failover
> > 
> > If qpid_heartbeat=60 and we're taking 2 minutes to fail over, that suggests
> > we're attempting to reconnect to the same host as our failed connection - it
> > could be an easy fix to cut the failover time in half
> 
> The way heartbeat is defined for AMQP 0-10 is that the connection should be
> considered failed if *two* heartbeat intervals pass without any  heartbeats.
> SO if you want to detect a failure within 60 seconds, the heartbeat interval
> should be 30secs.

Ah, that makes more sense (it pays to rtfm I guess).  So the 2m was only due to the timeout interval and had nothing to do with attempting to connect to the failed server (since the failure would be reported immediately)?  If so, the upstream bug is invalid and should be marked as such.


> 
> (Note also that the heartbeats are only sent if there is no other traffic on
> the connection).

(In reply to Maru Newby from comment #17)
> Ah, that makes more sense (it pays to rtfm I guess).  So the 2m was only due
> to the timeout interval and had nothing to do with attempting to connect to
> the failed server (since the failure would be reported immediately)?  If so,
> the upstream bug is invalid and should be marked as such.

With a heartbeat interval of 60 seconds, qpid will take 2 minutes to give up on a connection over which it receives no heartbeats.

I'm not familiar enough with all the various codebases to authoritatively say whether the reconnect logic in the driver is correct. However, looking at Connection.reconnect() in olso.messaging, it would appear that that will always try each of the defined brokers, starting always with the first in the list[1] and waiting for an increasing delay between failures (delay starts at 1 second and doubles up to a maximum of 60 seconds).

The nova.openstack.common.rpc.impl_qpid.py appears to have the same logic in it also.

So to me, https://bugs.launchpad.net/oslo.messaging/+bug/1261631 appears to accurately describe an issue.

In the original description of this bz above, I'm not clear from the log snippets what exactly happens for the first failure. It looks like on reconnect the first broker is tried and fails, so (following a 1 second sleep) it tries and succeeds to connect to the second. The second failure is then detected in 2 minutes (which is the time expected for a failure detected due to lack of heartbeats with a heartbeat interval of 60, however the error is 'connection aborted' whereas I would have expected 'heartbeat timeout'). The next reconnect starts (as always) from the first broker (which is now available again, and happens to be correct for this test).

Note also that I discovered a defect in the qpid python client, where the heartbeat only works for established connections and would not ensure that a connection attempt would timeout: https://issues.apache.org/jira/browse/QPID-5428.

[1]
   attempt = 0
   ...
   while True:
       ...
       broker = self.brokers[attempt % len(self.brokers)]

There are several problems mentioned here:
1. oslo.messaging qpid implementation does not do any load-balancing (so there is only one connection each moment of time, messages are not balanced thru qpid cluster). This should be discussed whether we really want to load-balance (not just failover) qpid messages.
2. 'connection aborted' instead of expected 'timeout'. This is hidden inside qpid python bindings. See: /usr/lib/python2.7/site-packages/qpid/connection.py. I should investigate more on why we don't get timeout error if heartbeat timeout is triggered.
3. On failover, we always restart from the start of broker list. Same as: https://bugs.launchpad.net/oslo.messaging/+bug/1261631. I have a fix for oslo-incubator for this (qpid part only), I'll handle review process for this. I'm curious whether we want to also update rabbitmq implementation.
4. timeout detection takes 2 mins which is too long. This should be fixed by setting proper value for qpid_heartbeat and probably does not require any change in default configuration file (?)

On problem 2. from comment #19: I actually get the following traceback when heartbeat fails, so there is probably no problem here:

2014-01-21 12:40:43.845 14523 ERROR neutron.openstack.common.rpc.impl_qpid [-] Failed to consume message from queue: heartbeat timeout
2014-01-21 12:40:43.845 14523 TRACE neutron.openstack.common.rpc.impl_qpid Traceback (most recent call last):
2014-01-21 12:40:43.845 14523 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 527, in ensure
2014-01-21 12:40:43.845 14523 TRACE neutron.openstack.common.rpc.impl_qpid     return method(*args, **kwargs)
2014-01-21 12:40:43.845 14523 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 584, in _consume
2014-01-21 12:40:43.845 14523 TRACE neutron.openstack.common.rpc.impl_qpid     nxt_receiver = self.session.next_receiver(timeout=timeout)
2014-01-21 12:40:43.845 14523 TRACE neutron.openstack.common.rpc.impl_qpid   File "<string>", line 6, in next_receiver
2014-01-21 12:40:43.845 14523 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 660, in next_receiver
2014-01-21 12:40:43.845 14523 TRACE neutron.openstack.common.rpc.impl_qpid     if self._ecwait(lambda: self.incoming, timeout):
2014-01-21 12:40:43.845 14523 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 50, in _ecwait
2014-01-21 12:40:43.845 14523 TRACE neutron.openstack.common.rpc.impl_qpid     result = self._ewait(lambda: self.closed or predicate(), timeout)
2014-01-21 12:40:43.845 14523 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 566, in _ewait
2014-01-21 12:40:43.845 14523 TRACE neutron.openstack.common.rpc.impl_qpid     result = self.connection._ewait(lambda: self.error or predicate(), timeout)
2014-01-21 12:40:43.845 14523 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 209, in _ewait
2014-01-21 12:40:43.845 14523 TRACE neutron.openstack.common.rpc.impl_qpid     self.check_error()
2014-01-21 12:40:43.845 14523 TRACE neutron.openstack.common.rpc.impl_qpid   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 202, in check_error
2014-01-21 12:40:43.845 14523 TRACE neutron.openstack.common.rpc.impl_qpid     raise self.error
2014-01-21 12:40:43.845 14523 TRACE neutron.openstack.common.rpc.impl_qpid HeartbeatTimeout: heartbeat timeout

pushing for A3 as this is HA related and HA would be delivered in A3. In addition we have u/s issues with getting the patch merged -
http://post-office.corp.redhat.com/archives/rh-openstack-dev/2014-January/msg00675.html

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2014-0516.html