Bug 1167414 - Configure TCP keepalive setting to all nodes (compute & controller nodes)
Summary: Configure TCP keepalive setting to all nodes (compute & controller nodes)
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-foreman-installer
Version: 5.0 (RHEL 7)
Hardware: Unspecified
OS: Unspecified
high
urgent
Target Milestone: ga
: Installer
Assignee: Jason Guiditta
QA Contact: Alexander Chuzhoy
URL:
Whiteboard:
Depends On:
Blocks: 1129242 1171744 1651230
TreeView+ depends on / blocked
 
Reported: 2014-11-24 17:00 UTC by Miguel Angel Ajo
Modified: 2019-04-16 14:25 UTC (History)
17 users (show)

Fixed In Version: openstack-foreman-installer-3.0.8-1.el7ost
Doc Type: Bug Fix
Doc Text:
Some services sustained stale TCP connections to RabbitMQ, which caused instability and connection failures when RabbitMQ was unavailable. This fix adds stronger TCP keepalive settings to the system configuration, which reduces the chance of system failure.
Clone Of:
: 1171744 (view as bug list)
Environment:
Last Closed: 2015-02-09 15:15:58 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2015:0156 normal SHIPPED_LIVE Red Hat Enterprise Linux OpenStack Platform Installer Bug Fix Advisory 2015-02-09 20:13:39 UTC

Description Miguel Angel Ajo 2014-11-24 17:00:30 UTC
Description of problem:
There are several bugs which prevent proper reconnection when vip-rabbitmq is moved or the rabbitmq server is gone. That makes many rabbitmq queues stuck.

https://bugzilla.redhat.com/show_bug.cgi?id=1129242


/etc/sysctl.d/tcp_keepalive.conf must be set to:
net.ipv4.tcp_keepalive_intvl = 1
net.ipv4.tcp_keepalive_probes = 5
net.ipv4.tcp_keepalive_time = 5

and this must be executed before starting computenode, nova-conductor or neutron-server (if we don't reboot after setting the previous file):

sysctl net.ipv4.tcp_keepalive_intvl=1
sysctl net.ipv4.tcp_keepalive_probes=5
sysctl net.ipv4.tcp_keepalive_time=5


/etc/haproxy/haproxy.cfg must include the following option in:

frontend vip-rabbitmq
  option tcpka
..
..
..

backend rabbitmq-vms   (or the normal o-f-i one)
  option tcpka
..
..
..

Comment 4 Ivan Chavero 2014-11-25 00:46:16 UTC
I have been checking how to approach this bug and the solution proposed here does not fit into a specific puppet module.
The puppet modules deal with general functionalities and it's up to the manifest creator (staypuft in this case) to use them in a specific environment.

Changing the component to OFI

Comment 5 Crag Wolfe 2014-11-25 00:53:43 UTC
From the OFI/quickstack perspective, both sysctl settings and haproxy settings may be added here:

https://github.com/redhat-openstack/astapor/tree/master/puppet/modules/quickstack/manifests/pacemaker

Comment 6 Alvaro Lopez Ortega 2014-11-25 12:01:04 UTC
This issue also impacts Packstack deployments though. What's the plan to fix it in those?

Comment 7 Ivan Chavero 2014-11-25 12:07:38 UTC
We can do this changes on packstack.

Comment 8 Miguel Angel Ajo 2014-11-25 12:42:06 UTC
There is piece which was missing in the initial comment
from the discussion on this bug: 
https://bugzilla.redhat.com/show_bug.cgi?id=1129242#c9

It needs to be added to the /etc/rabbitmq/rabbitmq.config

To make sure rabbitmq servers listen with tcp keepalive option
enabled. This protect the case when a node holding the vip-rabbitmq
is gone, making the tcp connections from haproxy(gone) to rabbitmq
get closed.

[
  {rabbit, [
    {cluster_nodes, {['rabbit@rhos5-node1', 'rabbit@rhos5-node2', 'rabbit@rhos5-node3'], disc}},
    {default_user, <<"guest">>},
    {default_pass, <<"guest">>},
+    {tcp_listen_options, [binary,
+                                {packet, raw},
+                                {reuseaddr, true},
+                                {backlog, 128},
+                                {nodelay, true},
+                                {exit_on_close, false},
+                                {keepalive, true}]}
+  ]},
  {kernel, [
        {inet_dist_listen_min, 9100},{inet_dist_listen_max, 9200}
  ]}
].

Comment 9 Ryan O'Hara 2014-11-25 14:52:59 UTC
(In reply to Alvaro Lopez Ortega from comment #6)
> This issue also impacts Packstack deployments though. What's the plan to fix
> it in those?

How? This seems completely related to HA deployments, which packstack does not do.

Comment 10 Ryan O'Hara 2014-11-25 14:55:54 UTC
(In reply to Miguel Angel Ajo from comment #0)
> Description of problem:
> There are several bugs which prevent proper reconnection when vip-rabbitmq
> is moved or the rabbitmq server is gone. That makes many rabbitmq queues
> stuck.
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1129242
> 
> 
> /etc/sysctl.d/tcp_keepalive.conf must be set to:
> net.ipv4.tcp_keepalive_intvl = 1
> net.ipv4.tcp_keepalive_probes = 5
> net.ipv4.tcp_keepalive_time = 5
> 
> and this must be executed before starting computenode, nova-conductor or
> neutron-server (if we don't reboot after setting the previous file):
> 
> sysctl net.ipv4.tcp_keepalive_intvl=1
> sysctl net.ipv4.tcp_keepalive_probes=5
> sysctl net.ipv4.tcp_keepalive_time=5
> 
> 
> /etc/haproxy/haproxy.cfg must include the following option in:
> 
> frontend vip-rabbitmq
>   option tcpka
> ..
> ..
> ..
> 
> backend rabbitmq-vms   (or the normal o-f-i one)
>   option tcpka
> ..
> ..
> ..

Use 'option clitcpka' in the frontend and 'option srvtcpka' in the backend. See haproxy documentation for details.

Comment 11 Miguel Angel Ajo 2014-11-25 15:23:53 UTC
> Use 'option clitcpka' in the frontend and 'option srvtcpka' in the backend.
> See haproxy documentation for details.

As far as I understood from haproxy documentation, both options should be equivalent, just tcpka works as "clitcpka" in the frontend and "srvtcpka" at the backend, or did I get it wrong?.

Comment 12 Miguel Angel Ajo 2014-11-25 15:28:13 UTC
The keepalive settings (rabbitmq, and sysctl) could be used for non-HA deployments, to handle the case where rabbitmq is installed on a separate host, and the host goes away (reboot -f for testing).

In that case, controllers and agents could have stale connections to the "old rabbit" waiting for messages up to 7200 seconds (2 hours).

Comment 13 Ryan O'Hara 2014-11-25 15:57:48 UTC
(In reply to Miguel Angel Ajo from comment #11)
> > Use 'option clitcpka' in the frontend and 'option srvtcpka' in the backend.
> > See haproxy documentation for details.
> 
> As far as I understood from haproxy documentation, both options should be
> equivalent, just tcpka works as "clitcpka" in the frontend and "srvtcpka" at
> the backend, or did I get it wrong?.

"Using option "tcpka" enables the emission of TCP keep-alive probes on both
the client and server sides of a connection. Note that this is meaningful
only in "defaults" or "listen" sections. If this option is used in a
frontend, only the client side will get keep-alives, and if this option is
used in a backend, only the server side will get keep-alives. For this
reason, it is strongly recommended to explicitly use "option clitcpka" and
"option srvtcpka" when the configuration is split between frontends and
backends."

Just do this.

Comment 14 Miguel Angel Ajo 2014-11-25 16:23:02 UTC
(In reply to Ryan O'Hara from comment #13)
> (In reply to Miguel Angel Ajo from comment #11)
> > > Use 'option clitcpka' in the frontend and 'option srvtcpka' in the backend.
> > > See haproxy documentation for details.
> > 
> > As far as I understood from haproxy documentation, both options should be
> > equivalent, just tcpka works as "clitcpka" in the frontend and "srvtcpka" at
> > the backend, or did I get it wrong?.
> 
> "Using option "tcpka" enables the emission of TCP keep-alive probes on both
> the client and server sides of a connection. Note that this is meaningful
> only in "defaults" or "listen" sections. If this option is used in a
> frontend, only the client side will get keep-alives, and if this option is
> used in a backend, only the server side will get keep-alives. For this
> reason, it is strongly recommended to explicitly use "option clitcpka" and
> "option srvtcpka" when the configuration is split between frontends and
> backends."
> 
> Just do this.

Ack, so no behavioral change, just the general recommendation
to be more explicit.

I'm updating the DocString with that, for reference:

/etc/haproxy/haproxy.cfg must include the following option in:

frontend vip-rabbitmq
  option clitcpka
..
..
..

backend rabbitmq-vms   (or the normal o-f-i one)
  option srvtcpka
..
..
..

Comment 15 Crag Wolfe 2014-11-25 16:59:29 UTC
Ryan, this is what we currently have in ofi/HA /etc/haproxy/haproxy.cfg

listen amqp
  bind 192.168.201.13:5672
  mode  tcp
  option  tcplog
  stick  on dst
  stick-table  type ip size 2
  timeout  client 120s
  timeout  server 120s
  server pcmk-10 192.168.200.10:15672  check inter 1s
  server pcmk-20 192.168.200.20:15672  check inter 1s
  server pcmk-30 192.168.200.30:15672  check inter 1s

You are saying we just need to s/option  tcplog/option  tcplog clitcpka/ above, right?

Comment 16 Crag Wolfe 2014-11-25 17:03:03 UTC
Should have made that 2 separate lines as in:

 option tcplog
 option clitcpka

Comment 17 Miguel Angel Ajo 2014-11-25 17:07:40 UTC
(In reply to Crag Wolfe from comment #15)
> Ryan, this is what we currently have in ofi/HA /etc/haproxy/haproxy.cfg
> 
> listen amqp
>   bind 192.168.201.13:5672
>   mode  tcp
>   option  tcplog
>   stick  on dst
>   stick-table  type ip size 2
>   timeout  client 120s
>   timeout  server 120s
>   server pcmk-10 192.168.200.10:15672  check inter 1s
>   server pcmk-20 192.168.200.20:15672  check inter 1s
>   server pcmk-30 192.168.200.30:15672  check inter 1s
> 
> You are saying we just need to s/option  tcplog/option  tcplog clitcpka/
> above, right?

Not sure, 

I'm not experienced with haproxy, but it seems like you are mixing frontend and backend in the same section?, for that may be you need to use 

option tcpka

Which actually covers the bind/and server side.

Ryan?

Comment 18 Ryan O'Hara 2014-11-25 18:07:33 UTC
(In reply to Crag Wolfe from comment #15)
> Ryan, this is what we currently have in ofi/HA /etc/haproxy/haproxy.cfg
> 
> listen amqp
>   bind 192.168.201.13:5672
>   mode  tcp
>   option  tcplog
>   stick  on dst
>   stick-table  type ip size 2
>   timeout  client 120s
>   timeout  server 120s
>   server pcmk-10 192.168.200.10:15672  check inter 1s
>   server pcmk-20 192.168.200.20:15672  check inter 1s
>   server pcmk-30 192.168.200.30:15672  check inter 1s
> 
> You are saying we just need to s/option  tcplog/option  tcplog clitcpka/
> above, right?

No. This is being complicated by talk of frontend/backend syntax while staypuft/OFI is using listen blocks. Since we're deploying with listen blocks, use 'option tcpka'. Anyone using fontend/backend style configuration should use clitcpka and srvtcpka.

So for 'listen amqp' you want to simply add 'option tcpka'.

The haproxy [1] documentation explains this quite well.

[1] http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#4-option%20tcpka

Comment 19 Jason Guiditta 2014-11-25 21:45:32 UTC
Merged and built.  Note that the rabbit config part appears to need changes to puppet-rabbitmq, which quickstack would then get for free once added, so we may need to clone this BZ to opm to cover that side.

Comment 22 Jason Guiditta 2014-12-08 14:13:57 UTC
Forgot to do this, just cloned and added some extra detail to filing text:

https://bugzilla.redhat.com/show_bug.cgi?id=1171744

Comment 23 Jason Guiditta 2014-12-16 22:25:13 UTC
OK, after discussion with the packstack team, we are going to replicate what they have done, but setting the tcp_keepalive = false and instead overriding with 
config_variables    => {'tcp_listen_options'  => "[binary,{packet, raw},{reuseaddr, true},{backlog, 128},{nodelay, true},{exit_on_close, false},{keepalive, true}]"}

Comment 24 Jason Guiditta 2014-12-16 22:27:51 UTC
Reference to packstack code:
https://review.openstack.org/#/c/137097/

Comment 25 Emilien Macchi 2014-12-17 13:58:03 UTC
As far I know we only need this feature for loadbalancer nodes, while the title indicates "controllers + computes". Am I missing something?

Comment 26 Jason Guiditta 2014-12-17 15:24:33 UTC
Patch posted for rabbit side:
https://github.com/redhat-openstack/astapor/pull/437

Emilien - the systctl part of this patch was request to be applied on both compute and control, that is why they are both listed in the subject.

Comment 27 Jason Guiditta 2014-12-17 18:11:30 UTC
Better version from cwolfe:
https://github.com/redhat-openstack/astapor/pull/440

Comment 28 Jason Guiditta 2014-12-17 20:08:49 UTC
Merged

Comment 34 Alexander Chuzhoy 2015-01-21 17:03:13 UTC
Verified:   FailedQA
Environment:
ruby193-rubygem-foreman_openstack_simplify-0.0.6-8.el7ost.noarch
openstack-foreman-installer-3.0.10-2.el7ost.noarch
ruby193-rubygem-staypuft-0.5.12-1.el7ost.noarch
rhel-osp-installer-client-0.5.5-2.el7ost.noarch
openstack-puppet-modules-2014.2.8-1.el7ost.noarch
rhel-osp-installer-0.5.5-2.el7ost.noarch


1.
Running "grep -i rabbit /etc/haproxy/haproxy.cfg" on controllers doesn't match anything.

2. /etc/sysctl.d/tcp_keepalive.conf doesn't exist.

/usr/lib/sysctl.d/00-system.conf exists instead and it has these entries:
net.ipv4.ip_nonlocal_bind=1
net.ipv4.tcp_keepalive_intvl=1
net.ipv4.tcp_keepalive_time=5
net.ipv4.tcp_keepalive_probes=5

Comment 35 Ryan O'Hara 2015-01-21 17:11:36 UTC
(In reply to Alexander Chuzhoy from comment #34)
> 1.
> Running "grep -i rabbit /etc/haproxy/haproxy.cfg" on controllers doesn't
> match anything.

I believe the proxy is named 'amqp', not rabbit.

Comment 38 Jason Guiditta 2015-01-21 21:42:04 UTC
this is what a quickstack-configured haproxy.cfg contains:

listen amqp
  bind 192.168.201.13:5672
  mode  tcp
  option  tcpka
  option  tcplog
  timeout  client 900m
  timeout  server 900m
  server pcmk-c1a1 192.168.200.10:5672  check inter 1s
  server pcmk-c1a2 192.168.200.20:5672  check inter 1s
  server pcmk-c1a3 192.168.200.30:5672  check inter 1s


This is correct as far as I know

Comment 39 Miguel Angel Ajo 2015-01-22 13:15:12 UTC
Looking at the changes it seems good for OFI.

My suggestions were based on the OSP6 HA ref-arch, but they are equivalent.

I will help sasha verify the deployment with this.

Comment 41 Alexander Chuzhoy 2015-01-23 15:37:32 UTC
Verified:

Environment:

ruby193-rubygem-foreman_openstack_simplify-0.0.6-8.el7ost.noarch
openstack-foreman-installer-3.0.10-2.el7ost.noarch
ruby193-rubygem-staypuft-0.5.14-1.el7ost.noarch
rhel-osp-installer-client-0.5.5-2.el7ost.noarch
openstack-puppet-modules-2014.2.8-1.el7ost.noarch
rhel-osp-installer-0.5.5-2.el7ost.noarch



All the expected settings were found in these files:
/etc/rabbitmq/rabbitmq.config
/etc/haproxy/haproxy.cfg
/etc/sysctl.d/99-sysctl.conf

Comment 43 errata-xmlrpc 2015-02-09 15:15:58 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-0156.html


Note You need to log in before you can comment on or make changes to this bug.