Bug 1826053 - [IPI][Baremetal] Keepalived container stopped working after applying node's networking configuration change
Summary: [IPI][Baremetal] Keepalived container stopped working after applying node's ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Machine Config Operator
Version: 4.4
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.5.0
Assignee: Yossi Boaron
QA Contact: Victor Voronkov
URL:
Whiteboard:
: 1851447 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-04-20 19:02 UTC by Yossi Boaron
Modified: 2023-10-06 19:42 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Updating node's networking post-deployment (for example by using kind: MachineConfig) in such a way that the interface that was originally carrying VIPs is attached to a bridge (OVS or Linux bridge). Consequence: While the Keepalived-monitor observed the change and rendered new config, Keepalived container failed with 'permanent error CONFIG' and not being restarted by Kubelet. Fix: Update Liveness probe of Keepalived container to check also Keepalived process existence. Result: If for any reason the Keepalived process exits, Kubelet will detect this and restart the Keepalived container.
Clone Of:
Environment:
Last Closed: 2020-07-13 17:29:11 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift machine-config-operator pull 1604 0 None closed [baremetal] Update keepalived Liveness check 2020-11-11 09:27:25 UTC
Red Hat Product Errata RHBA-2020:2409 0 None None None 2020-07-13 17:29:31 UTC

Description Yossi Boaron 2020-04-20 19:02:09 UTC
Description of problem:

This use case was reported by Petr Horacek.

Use case details:
I reconfigured the host by running [A], so the interface that was originally carrying VIPs is attached to an OVS bridge.

While the keepalived-monitor observed the change and rendered new config, keepalived container failed with 'permanent error CONFIG' and not being restarted by Kubelet.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. Stop/ kill the keepalives process inside keepalived container (use sudo crictl)
2. Using *sudo crictl ps* and the keepalived logs, verify the container was restarted properly by kubelet.




Actual results:
Keepalived container stopped working (error CONFIG) and kubelet doesn't restart the container.


Expected results:
keepalived-monitor should render a new keepalived config file.
Keepalived should be restarted, and apply the new config.

[A] https://access.redhat.com/articles/4934131

Comment 2 Eldar Weiss 2020-07-05 07:58:25 UTC
Please add Reproduction steps as [A] link is unavailable.

Comment 3 Yossi Boaron 2020-07-05 16:34:50 UTC
You can stop/kill the keepalived process inside the Kepelaived container and verify that the container is restarted properly by Kubelet.

Comment 6 errata-xmlrpc 2020-07-13 17:29:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409

Comment 7 Yossi Boaron 2020-09-09 13:46:06 UTC
*** Bug 1851447 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.