Bug 1445905
Summary: | On IPv6 deployments the VIP is configured as /64 and can be used as source address by services on the node | ||
---|---|---|---|
Product: | Red Hat OpenStack | Reporter: | Michele Baldessari <michele> |
Component: | puppet-tripleo | Assignee: | Michele Baldessari <michele> |
Status: | CLOSED ERRATA | QA Contact: | pkomarov |
Severity: | high | Docs Contact: | |
Priority: | high | ||
Version: | 11.0 (Ocata) | CC: | ccollett, chjones, fdinitto, jeckersb, jjoyce, jschluet, markmc, michele, mkrcmari, pmyers, royoung, rscarazz, slinaber, tvignaud |
Target Milestone: | z2 | Keywords: | Triaged, ZStream |
Target Release: | 11.0 (Ocata) | ||
Hardware: | All | ||
OS: | Linux | ||
Whiteboard: | |||
Fixed In Version: | puppet-tripleo-6.5.0-7.el7ost openstack-tripleo-heat-templates-6.2.0-2.el7ost | Doc Type: | Known Issue |
Doc Text: |
In Highly Available IPv6 deployments, virtual IPs used for RabbitMQ may move between controller hosts during an upgrade. A bug in the creation of these IPv6 IPs causes them to be used as source addresses for RabbitMQ's connections. As a result, RabbitMQ will crash and may be unable to automatically recover its cluster.
To return to normal operation, restart RabbitMQ on the affected controller hosts, as well as any services which depend on RabbitMQ and do not automatically reconnect.
|
Story Points: | --- |
Clone Of: | Environment: | ||
Last Closed: | 2017-09-13 21:43:17 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Bug Depends On: | 1445861 | ||
Bug Blocks: |
Description
Michele Baldessari
2017-04-26 18:17:44 UTC
We need fixes for both puppet-tripleo and puppet-pacemaker? Do we need a puppet-pacemaker bugzilla as well as this one for puppet-tripleo? Usually releng is able to make one errata of multiple packages with one BZ (at least in my experience). In this case we still need to sketch out how to bring this fix out with a) minor updates and b) major upgrades. (Hopefully only a) is needed but we need to first sketch out a full plan). I expect we will need tripleo-heat-template fixes as well for a) and/or b) Ok so here is a short status update. In order to fix this issue we have two scenarios: - New deployments For new deployments we need three patches (two for puppet-pacemaker and one for puppet-tripleo) A.1) puppet-pacemaker: https://review.openstack.org/460232 - Add support for ipv6_addrlabel with IPaddr2 RA A.2) puppet-pacemaker: https://review.openstack.org/462073 - Fix a typo in ipv6 addrlabel A.3) puppet-tripleo: https://review.openstack.org/460028 - IPv6 VIP addresses need to be /128 So puppet-pacemaker has no stable branches, A.1 and A.2 are the reviews we need. For puppet-tripleo the review for master at A.3 has merged. The backport is here: A.4) puppet-tripleo: https://review.openstack.org/462479 - IPv6 VIP addresses need to be /128 - Existing deployments B.1) tripleo-heat-templates: https://review.openstack.org/#/c/460724/ - Initial VIP ipv6 minor update code B.1 is the master review and has merged. The backport to ocata is here: B.2) tripleo-heat-templates: https://review.openstack.org/462480 - Initial VIP ipv6 minor update code I will link the backports only for tht and puppet-tripleo and the master reviews for puppet-pacemaker. Once the backports are merged, I will go over this with mburns and see if we need to split off a separate bz for puppet-pacemaker or not. FTR: The pin update for puppet-pacemaker in newton/ocata, which is needed to include the ipv6 fixes in puppet-pacemaker, has happened here: https://review.rdoproject.org/r/#/c/6519/ All four reviews (see comment#4) have no merged (two puppet-pacemaker ones for master branch and two for stable/ocata (tht and puppet-tripleo)). Moving to POST Bug verification steps taken: Check that we are on rhos-11 GA: [stack@undercloud ~]$ cat core_puddle_version 2017-05-09.2 [stack@undercloud ~]$ grep -v '\#' /etc/yum.repos.d/rhos-release-11.repo|grep -m 1 baseurl baseurl=http://download.lab.bos.redhat.com/rcm-guest/puddles/OpenStack/11.0-RHEL-7/2017-05-09.2/RH7-RHOS-11.0/$basearch/os [stack@undercloud ~]$curl -s http://download-node-02.eng.bos.redhat.com/rcm-guest/puddles/OpenStack/11.0-RHEL-7/|grep GA|cut -f 3,4 -d'=' "[DIR]"> <a href="GA/">GA/</a> 09-May-2017 10:15 - Check that we have, on the overcloud controllers /64 ipv6 vips: [root@controller-0 ~]# pcs status |grep `hostname -s`|grep 2620 ip-2620.52.0.23ae..16 (ocf::heartbeat:IPaddr2): Started controller-0 [root@controller-0 ~]# ip a show vlan189 14: vlan189: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1000 link/ether 1e:24:3e:8c:73:86 brd ff:ff:ff:ff:ff:ff inet6 2620:52:0:23ae::16/64 scope global After update from GA -> Z2 , check correct update to z2 : [stack@puma33 ~]$ cat core_puddle_version 2017-08-30.3 [stack@puma33 ~]$ grep -v '\#' /etc/yum.repos.d/rhos-release-11.repo|grep -m 1 baseurl baseurl=http://download.lab.bos.redhat.com/rcm-guest/puddles/OpenStack/11.0-RHEL-7/2017-08-30.3/RH7-RHOS-11.0/$basearch/os [stack@puma33 ~]$ curl -s http://download-node-02.eng.bos.redhat.com/rcm-guest/puddles/OpenStack/11.0-RHEL-7/|grep z2|cut -f 3,4 -d'=' "[DIR]"> <a href="z2/">z2/</a> 30-Aug-2017 23:29 Check that we now have , on the overcloud controllers /128 ipv6 vips: [root@controller-0 ~]# pcs status |grep `hostname -s`|grep 2620 ip-2620.52.0.23b4..15 (ocf::heartbeat:IPaddr2): Started controller-0 ip-2620.52.0.23ae..16 (ocf::heartbeat:IPaddr2): Started controller-0 [root@controller-0 ~]# ip a show vlan189 11: vlan189: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1000 link/ether 02:54:64:bd:07:fc brd ff:ff:ff:ff:ff:ff inet6 2620:52:0:23ae::16/128 scope global [root@controller-0 ~]# ip a show vlan195 12: vlan195: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1000 link/ether be:38:68:39:09:73 brd ff:ff:ff:ff:ff:ff inet6 2620:52:0:23b4::15/128 scope global Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:2721 |