Description of problem: The keepalive pod in one node always tries to be master, despite the master is already in another pod in another node, generating split brain. This is consistent on one node. Version-Release number of selected component (if applicable): 3.3.0 How reproducible: Unconfirmed Actual results: Two nodes working, when marking third node as schedulable (allowing a third ipfailover pod) it tries to take master. This leads to splitbrain and "no route to host" messages when trying to use routing to access services. Expected results: High availability setup working, single master Additional info: The behavior we are seeing (see following comments/uploads) is normally caused when there is something blocking connection between the nodes. However we have verified that the iptables look fine and that no other firewall is blocking the nodes. Multicast should be working. Yet they are not communicating properly.
Can we get the full iptables output from the "bad" node, and the 'oc logs' output from the three ipfailover pods?
See: https://github.com/openshift/origin/pull/12399 out for review
Commit pushed to master at https://github.com/openshift/origin https://github.com/openshift/origin/commit/11034c6fc7fbe50d6cf9442c47749acf77407a87 ipfailover keepalived split brain All of the VIPs try to end up on the same node. The process doesn't settle on a node. This fix eliminates the vrrp_sync_group. The result is each VIP settles on a node and not all appear on the same node. There is some general coding style cleanup included as well. The check and notify scripts are in the vrrp_instance for each VIP. Note: This may look like a script generating a script. It is not. The script generates the keepalived.conf config file which is input by keepalived when it starts. Bug 1402536, 1405015 https://bugzilla.redhat.com/show_bug.cgi?id=1402536 https://bugzilla.redhat.com/show_bug.cgi?id=1405015 Signed-off-by: Phil Cameron <pcameron>
*** Bug 1405015 has been marked as a duplicate of this bug. ***
per c#14, could you please decide if you will backport the fix to 3.4/3.3? (although it may have the workaround as below) 3.2 -> works well (no problem) 3.3 -> has this bug (use 3.2 image for the workaround) 3.4 -> has this bug (use 3.2 image for the workaround)
The 3.5 version is quite different than earlier versions. What do you think about forward porting the change in images/ipfailover/keepalived/lib/config-generators.sh 3.2 into 3.3, 3.4? It amounts to reverting a one line change. -$(generate_vrrp_sync_groups "$HA_CONFIG_NAME" "$vips") << 3.2 +$(generate_vrrp_sync_groups "$HA_CONFIG_NAME" "$HA_VIPS") << 3.3, 3.4
This bug needs to be cloned so we can have it for 3.3, 3.4 and 3.5.
@pcameron: Can you clone it, set the target release appropriately and link the right PRs to each bug and put them in MODIFIED state please?
Cloned for OSE 3.3 version https://bugzilla.redhat.com/show_bug.cgi?id=1417276
Cloned for OSE 3.4 version https://bugzilla.redhat.com/show_bug.cgi?id=1417282
This has been merged into ocp and is in OCP v3.5.0.12 or newer.
@troy Dawson I checked this using keepalived image (openshift3/ose-keepalived-ipfailover v3.3.1.12 97b4d028d09c 2 days ago) # oc rsh ipf-1-36g9o cat keepalived/lib/config-generators.sh <------snip----> 249 $(generate_vrrp_sync_groups "$HA_CONFIG_NAME" "$HA_VIPS") <-----snip-----> Could you help the image if has been rebuilt using the latest code. since I saw the repo code had been updated at below. https://github.com/openshift/ose/blob/enterprise-3.3/images/ipfailover/keepalived/lib/config-generators.sh#L249
Move the bug to ON_QA since the 3.3 issue is tracking by another bug.
Tested with OCP build 3.5.0.16. After scale the ipfailover dc up, the new pod will get the MASTER state, and the old pod will enter BACKUP state. Move the bug to verified.
As I was hit by this bug on OSE 3.2, will there be a fix for 3.2 images also? The Redhat CASE was 01745024.
3.2 was reported as working. I see youare using 3.2.1. Can you roll back the ipfailover image to 3.2.0 to work around it?
Will there be an update for 3.2/3.3 versions of an image?
See clone 1417276 for 3.3 release. No change for 3.2 since the problem is not there and it is the workaround.
bbennett 3.2.1.26 has the fix - last changed 2015-04-13
Sorry, Aleksander, I killed the needinfo. But we do need what Phil requested: Please get the image name from the ipfailover dc. Please get a copy of lib/config-generators.sh from the pod that is having problems. I just need lines 240-260 For the second, 'oc rsh <podname>' then grab: /var/lib/ipfailover/keepalived/lib/config-generators.sh You can do: grep generate_vrrp_sync_groups /var/lib/ipfailover/keepalived/lib/config-generators.sh
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:0884
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days