Description of problem: There are several bugs which prevent proper reconnection when vip-rabbitmq is moved or the rabbitmq server is gone. That makes many rabbitmq queues stuck. https://bugzilla.redhat.com/show_bug.cgi?id=1129242 /etc/sysctl.d/tcp_keepalive.conf must be set to: net.ipv4.tcp_keepalive_intvl = 1 net.ipv4.tcp_keepalive_probes = 5 net.ipv4.tcp_keepalive_time = 5 and this must be executed before starting computenode, nova-conductor or neutron-server (if we don't reboot after setting the previous file): sysctl net.ipv4.tcp_keepalive_intvl=1 sysctl net.ipv4.tcp_keepalive_probes=5 sysctl net.ipv4.tcp_keepalive_time=5 /etc/haproxy/haproxy.cfg must include the following option in: frontend vip-rabbitmq option tcpka .. .. .. backend rabbitmq-vms (or the normal o-f-i one) option tcpka .. .. ..
I have been checking how to approach this bug and the solution proposed here does not fit into a specific puppet module. The puppet modules deal with general functionalities and it's up to the manifest creator (staypuft in this case) to use them in a specific environment. Changing the component to OFI
From the OFI/quickstack perspective, both sysctl settings and haproxy settings may be added here: https://github.com/redhat-openstack/astapor/tree/master/puppet/modules/quickstack/manifests/pacemaker
This issue also impacts Packstack deployments though. What's the plan to fix it in those?
We can do this changes on packstack.
There is piece which was missing in the initial comment from the discussion on this bug: https://bugzilla.redhat.com/show_bug.cgi?id=1129242#c9 It needs to be added to the /etc/rabbitmq/rabbitmq.config To make sure rabbitmq servers listen with tcp keepalive option enabled. This protect the case when a node holding the vip-rabbitmq is gone, making the tcp connections from haproxy(gone) to rabbitmq get closed. [ {rabbit, [ {cluster_nodes, {['rabbit@rhos5-node1', 'rabbit@rhos5-node2', 'rabbit@rhos5-node3'], disc}}, {default_user, <<"guest">>}, {default_pass, <<"guest">>}, + {tcp_listen_options, [binary, + {packet, raw}, + {reuseaddr, true}, + {backlog, 128}, + {nodelay, true}, + {exit_on_close, false}, + {keepalive, true}]} + ]}, {kernel, [ {inet_dist_listen_min, 9100},{inet_dist_listen_max, 9200} ]} ].
(In reply to Alvaro Lopez Ortega from comment #6) > This issue also impacts Packstack deployments though. What's the plan to fix > it in those? How? This seems completely related to HA deployments, which packstack does not do.
(In reply to Miguel Angel Ajo from comment #0) > Description of problem: > There are several bugs which prevent proper reconnection when vip-rabbitmq > is moved or the rabbitmq server is gone. That makes many rabbitmq queues > stuck. > > https://bugzilla.redhat.com/show_bug.cgi?id=1129242 > > > /etc/sysctl.d/tcp_keepalive.conf must be set to: > net.ipv4.tcp_keepalive_intvl = 1 > net.ipv4.tcp_keepalive_probes = 5 > net.ipv4.tcp_keepalive_time = 5 > > and this must be executed before starting computenode, nova-conductor or > neutron-server (if we don't reboot after setting the previous file): > > sysctl net.ipv4.tcp_keepalive_intvl=1 > sysctl net.ipv4.tcp_keepalive_probes=5 > sysctl net.ipv4.tcp_keepalive_time=5 > > > /etc/haproxy/haproxy.cfg must include the following option in: > > frontend vip-rabbitmq > option tcpka > .. > .. > .. > > backend rabbitmq-vms (or the normal o-f-i one) > option tcpka > .. > .. > .. Use 'option clitcpka' in the frontend and 'option srvtcpka' in the backend. See haproxy documentation for details.
> Use 'option clitcpka' in the frontend and 'option srvtcpka' in the backend. > See haproxy documentation for details. As far as I understood from haproxy documentation, both options should be equivalent, just tcpka works as "clitcpka" in the frontend and "srvtcpka" at the backend, or did I get it wrong?.
The keepalive settings (rabbitmq, and sysctl) could be used for non-HA deployments, to handle the case where rabbitmq is installed on a separate host, and the host goes away (reboot -f for testing). In that case, controllers and agents could have stale connections to the "old rabbit" waiting for messages up to 7200 seconds (2 hours).
(In reply to Miguel Angel Ajo from comment #11) > > Use 'option clitcpka' in the frontend and 'option srvtcpka' in the backend. > > See haproxy documentation for details. > > As far as I understood from haproxy documentation, both options should be > equivalent, just tcpka works as "clitcpka" in the frontend and "srvtcpka" at > the backend, or did I get it wrong?. "Using option "tcpka" enables the emission of TCP keep-alive probes on both the client and server sides of a connection. Note that this is meaningful only in "defaults" or "listen" sections. If this option is used in a frontend, only the client side will get keep-alives, and if this option is used in a backend, only the server side will get keep-alives. For this reason, it is strongly recommended to explicitly use "option clitcpka" and "option srvtcpka" when the configuration is split between frontends and backends." Just do this.
(In reply to Ryan O'Hara from comment #13) > (In reply to Miguel Angel Ajo from comment #11) > > > Use 'option clitcpka' in the frontend and 'option srvtcpka' in the backend. > > > See haproxy documentation for details. > > > > As far as I understood from haproxy documentation, both options should be > > equivalent, just tcpka works as "clitcpka" in the frontend and "srvtcpka" at > > the backend, or did I get it wrong?. > > "Using option "tcpka" enables the emission of TCP keep-alive probes on both > the client and server sides of a connection. Note that this is meaningful > only in "defaults" or "listen" sections. If this option is used in a > frontend, only the client side will get keep-alives, and if this option is > used in a backend, only the server side will get keep-alives. For this > reason, it is strongly recommended to explicitly use "option clitcpka" and > "option srvtcpka" when the configuration is split between frontends and > backends." > > Just do this. Ack, so no behavioral change, just the general recommendation to be more explicit. I'm updating the DocString with that, for reference: /etc/haproxy/haproxy.cfg must include the following option in: frontend vip-rabbitmq option clitcpka .. .. .. backend rabbitmq-vms (or the normal o-f-i one) option srvtcpka .. .. ..
Ryan, this is what we currently have in ofi/HA /etc/haproxy/haproxy.cfg listen amqp bind 192.168.201.13:5672 mode tcp option tcplog stick on dst stick-table type ip size 2 timeout client 120s timeout server 120s server pcmk-10 192.168.200.10:15672 check inter 1s server pcmk-20 192.168.200.20:15672 check inter 1s server pcmk-30 192.168.200.30:15672 check inter 1s You are saying we just need to s/option tcplog/option tcplog clitcpka/ above, right?
Should have made that 2 separate lines as in: option tcplog option clitcpka
(In reply to Crag Wolfe from comment #15) > Ryan, this is what we currently have in ofi/HA /etc/haproxy/haproxy.cfg > > listen amqp > bind 192.168.201.13:5672 > mode tcp > option tcplog > stick on dst > stick-table type ip size 2 > timeout client 120s > timeout server 120s > server pcmk-10 192.168.200.10:15672 check inter 1s > server pcmk-20 192.168.200.20:15672 check inter 1s > server pcmk-30 192.168.200.30:15672 check inter 1s > > You are saying we just need to s/option tcplog/option tcplog clitcpka/ > above, right? Not sure, I'm not experienced with haproxy, but it seems like you are mixing frontend and backend in the same section?, for that may be you need to use option tcpka Which actually covers the bind/and server side. Ryan?
(In reply to Crag Wolfe from comment #15) > Ryan, this is what we currently have in ofi/HA /etc/haproxy/haproxy.cfg > > listen amqp > bind 192.168.201.13:5672 > mode tcp > option tcplog > stick on dst > stick-table type ip size 2 > timeout client 120s > timeout server 120s > server pcmk-10 192.168.200.10:15672 check inter 1s > server pcmk-20 192.168.200.20:15672 check inter 1s > server pcmk-30 192.168.200.30:15672 check inter 1s > > You are saying we just need to s/option tcplog/option tcplog clitcpka/ > above, right? No. This is being complicated by talk of frontend/backend syntax while staypuft/OFI is using listen blocks. Since we're deploying with listen blocks, use 'option tcpka'. Anyone using fontend/backend style configuration should use clitcpka and srvtcpka. So for 'listen amqp' you want to simply add 'option tcpka'. The haproxy [1] documentation explains this quite well. [1] http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#4-option%20tcpka
Merged and built. Note that the rabbit config part appears to need changes to puppet-rabbitmq, which quickstack would then get for free once added, so we may need to clone this BZ to opm to cover that side.
Forgot to do this, just cloned and added some extra detail to filing text: https://bugzilla.redhat.com/show_bug.cgi?id=1171744
OK, after discussion with the packstack team, we are going to replicate what they have done, but setting the tcp_keepalive = false and instead overriding with config_variables => {'tcp_listen_options' => "[binary,{packet, raw},{reuseaddr, true},{backlog, 128},{nodelay, true},{exit_on_close, false},{keepalive, true}]"}
Reference to packstack code: https://review.openstack.org/#/c/137097/
As far I know we only need this feature for loadbalancer nodes, while the title indicates "controllers + computes". Am I missing something?
Patch posted for rabbit side: https://github.com/redhat-openstack/astapor/pull/437 Emilien - the systctl part of this patch was request to be applied on both compute and control, that is why they are both listed in the subject.
Better version from cwolfe: https://github.com/redhat-openstack/astapor/pull/440
Merged
Verified: FailedQA Environment: ruby193-rubygem-foreman_openstack_simplify-0.0.6-8.el7ost.noarch openstack-foreman-installer-3.0.10-2.el7ost.noarch ruby193-rubygem-staypuft-0.5.12-1.el7ost.noarch rhel-osp-installer-client-0.5.5-2.el7ost.noarch openstack-puppet-modules-2014.2.8-1.el7ost.noarch rhel-osp-installer-0.5.5-2.el7ost.noarch 1. Running "grep -i rabbit /etc/haproxy/haproxy.cfg" on controllers doesn't match anything. 2. /etc/sysctl.d/tcp_keepalive.conf doesn't exist. /usr/lib/sysctl.d/00-system.conf exists instead and it has these entries: net.ipv4.ip_nonlocal_bind=1 net.ipv4.tcp_keepalive_intvl=1 net.ipv4.tcp_keepalive_time=5 net.ipv4.tcp_keepalive_probes=5
(In reply to Alexander Chuzhoy from comment #34) > 1. > Running "grep -i rabbit /etc/haproxy/haproxy.cfg" on controllers doesn't > match anything. I believe the proxy is named 'amqp', not rabbit.
this is what a quickstack-configured haproxy.cfg contains: listen amqp bind 192.168.201.13:5672 mode tcp option tcpka option tcplog timeout client 900m timeout server 900m server pcmk-c1a1 192.168.200.10:5672 check inter 1s server pcmk-c1a2 192.168.200.20:5672 check inter 1s server pcmk-c1a3 192.168.200.30:5672 check inter 1s This is correct as far as I know
Looking at the changes it seems good for OFI. My suggestions were based on the OSP6 HA ref-arch, but they are equivalent. I will help sasha verify the deployment with this.
Verified: Environment: ruby193-rubygem-foreman_openstack_simplify-0.0.6-8.el7ost.noarch openstack-foreman-installer-3.0.10-2.el7ost.noarch ruby193-rubygem-staypuft-0.5.14-1.el7ost.noarch rhel-osp-installer-client-0.5.5-2.el7ost.noarch openstack-puppet-modules-2014.2.8-1.el7ost.noarch rhel-osp-installer-0.5.5-2.el7ost.noarch All the expected settings were found in these files: /etc/rabbitmq/rabbitmq.config /etc/haproxy/haproxy.cfg /etc/sysctl.d/99-sysctl.conf
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2015-0156.html