Bug 1249661 - VIP is migrated between two controllers when third controller (not owning the VIP) is rebooted
VIP is migrated between two controllers when third controller (not owning the...
Status: CLOSED CURRENTRELEASE
Product: Red Hat OpenStack
Classification: Red Hat
Component: rhosp-director (Show other bugs)
Director
Unspecified Unspecified
medium Severity urgent
: ---
: 10.0 (Newton)
Assigned To: Hugh Brock
Shai Revivo
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-08-03 10:04 EDT by Asaf Hirshberg
Modified: 2016-08-31 00:46 EDT (History)
5 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-08-31 00:46:24 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
/var/log/pacemaker.log from where the reboot happend (813.12 KB, text/plain)
2015-08-03 10:04 EDT, Asaf Hirshberg
no flags Details

  None (edit)
Description Asaf Hirshberg 2015-08-03 10:04:35 EDT
Created attachment 1058776 [details]
/var/log/pacemaker.log from where the reboot happend

Description of problem:
When testing reboot to a controller without a VIP while running Rally traffic I saw that the vip(10.35.180.10) migrate between the two controllers that where active:

heat-admin@overcloud-controller-1 ~]$ sudo pcs status
Cluster name: tripleo_cluster
Last updated: Mon Aug  3 09:41:25 2015
Last change: Mon Aug  3 08:27:42 2015
Stack: corosync
Current DC: overcloud-controller-2 (3) - partition with quorum
Version: 1.1.12-a14efad
3 Nodes configured
112 Resources configured


Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]

Full list of resources:

 ip-172.17.0.11	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-1 
 Clone Set: haproxy-clone [haproxy]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 ip-172.18.0.10	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-2 
 ip-172.17.0.10	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-0 
 Master/Slave Set: galera-master [galera]
     Masters: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 ip-172.19.0.10	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-2 
 ip-10.35.180.10	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-1 


[heat-admin@overcloud-controller-1 ~]$ sudo pcs status
Cluster name: tripleo_cluster
Last updated: Mon Aug  3 09:47:55 2015
Last change: Mon Aug  3 08:27:42 2015
Stack: corosync
Current DC: overcloud-controller-1 (2) - partition with quorum
Version: 1.1.12-a14efad
3 Nodes configured
112 Resources configured


Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]

Full list of resources:

 ip-172.17.0.11	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-1 
 Clone Set: haproxy-clone [haproxy]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 ip-172.18.0.10	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-0 
 ip-172.17.0.10	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-2 
 Master/Slave Set: galera-master [galera]
     Masters: [ overcloud-controller-0 overcloud-controller-1 ]
     Slaves: [ overcloud-controller-2 ]
 ip-172.19.0.10	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-1 
 ip-10.35.180.10	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-0 


How reproducible:
3/3

Steps to Reproduce:
1.check with "pcs status" which controller hosts the VIP.
2.reboot the controller that doesn't host any vip.
3.check with "pcs status" if the VIP been migrated.
Comment 3 Asaf Hirshberg 2015-08-04 03:15:14 EDT

Description of problem:
vip(10.35.180.10) migrate between controller-1 and controller-0 when controller-2 been rebooted:
before the reboot:

heat-admin@overcloud-controller-1 ~]$ sudo pcs status
Last updated: Mon Aug  3 09:41:25 2015
Last change: Mon Aug  3 08:27:42 2015

Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]

Full list of resources:

 ip-172.17.0.11	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-1 
 Clone Set: haproxy-clone [haproxy]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 ip-172.18.0.10	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-2 
 ip-172.17.0.10	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-0 
controller-2 ]
 ip-172.19.0.10	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-2 
=> ip-10.35.180.10	(ocf::heartbeat:IPaddr2):	Started overcloud---controller-1 

[heat-admin@overcloud-controller-1 ~]$ sudo pcs status
Last updated: Mon Aug  3 09:47:55 2015
Last change: Mon Aug  3 08:27:42 2015

Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]

Full list of resources:

 ip-172.17.0.11	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-1 
 Clone Set: haproxy-clone [haproxy]
     Started: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
 ip-172.18.0.10	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-0 
 ip-172.17.0.10	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-2 
 ip-172.19.0.10	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-1 
--> ip-10.35.180.10	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-0 

This condition lead to downtime as the VIP(10.35.180.10) become unavailable although controller-1 and controller-0 are ok.  

How reproducible:
3/3

Steps to Reproduce:
1.check with "pcs status" which controller hosts the VIP.
2.reboot the controller that doesn't host any vip.
3.check with "pcs status" if the VIP been migrated.
Comment 5 Mike Burns 2016-04-07 16:47:27 EDT
This bug did not make the OSP 8.0 release.  It is being deferred to OSP 10.

Note You need to log in before you can comment on or make changes to this bug.