Bug 1920282
Summary: | kubelet bound to incorrect nic causing change in the IPs | ||
---|---|---|---|
Product: | OpenShift Container Platform | Reporter: | Vladislav Walek <vwalek> |
Component: | Networking | Assignee: | Alexander Constantinescu <aconstan> |
Networking sub component: | openshift-sdn | QA Contact: | zhaozhanqi <zzhao> |
Status: | CLOSED NOTABUG | Docs Contact: | |
Severity: | high | ||
Priority: | high | CC: | aconstan, aos-bugs, arghosh, bshirren, dmoessne, fshaikh, jorge.martinezgarcia, malonso, mfuruta, mvardhan, pawankum, rh-container, rphillips, smozowei, suchaudh, tsweeney, vpagar, ychoukse |
Version: | 4.5 | Keywords: | Reopened |
Target Milestone: | --- | Flags: | vwalek:
needinfo-
|
Target Release: | 4.8.0 | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | If docs needed, set a value | |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2021-03-22 08:04:21 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Vladislav Walek
2021-01-26 00:01:44 UTC
Could you please provide an update on this ? The Kubelet's node IP can be pinned to an IP by creating a file like this, and then rebooting afterwards: cat <<EOF > /etc/systemd/system/kubelet.service.d/20-nodenet.conf [Service] Environment="KUBELET_NODE_IP=10.0.1.2" EOF You will need to reboot the node after creating the file. (In reply to Ryan Phillips from comment #4) > The Kubelet's node IP can be pinned to an IP by creating a file like this, > and then rebooting afterwards: > > cat <<EOF > /etc/systemd/system/kubelet.service.d/20-nodenet.conf > [Service] > Environment="KUBELET_NODE_IP=10.0.1.2" > EOF > > You will need to reboot the node after creating the file. Great, I will KCS that. Hello Team, nodeip-configuration service is responsible to set KUBELET_NODE_IP in /etc/systemd/system/kubelet.service.d/20-nodenet.conf file. The kubelet.service reads KUBELET_NODE_IP from /etc/systemd/system/kubelet.service.d/20-nodenet.conf file. As part of nodeip-configuration service one container gets created and a script gets executed within the container to extract node IP. ~~~ ExecStart=/bin/bash -c " \ until \ /usr/bin/podman run --rm \ --authfile /var/lib/kubelet/config.json \ --net=host \ --volume /etc/systemd/system:/etc/systemd/system:z \ quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c0e6c751d3f42c280624f88bc91e1e27fa602a09fa57a41d0c34095d853e9e98 \ node-ip \ set --retry-on-failure; \ do \ sleep 5; \ done" ~~~ node IP is determined as per following rules. 1. the NIC is configured as the default gateway.[0] 2. the IP of the NIC is resolved as the node hostname. when 1 and 2 are satisfied, then that NIC IP would be set as the Openshift node IP. [0] [https://github.com/openshift/sdn/blob/7106dab9c5785ae163237d443de0ddc19cb1f040/vendor/k8s.io/apimachinery/pkg/util/net/interface.go#L395-L398] ~~~ // chooseHostInterfaceFromRoute cycles through each default route provided, looking for a // global IP address from the interface for the route. addressFamilies determines whether it // prefers IPv4 or IPv6 func chooseHostInterfaceFromRoute(routes []Route, nw networkInterfacer, addressFamilies AddressFamilyPreference) ~~~ If you run below command on any node, you will always see the node IP is chosen as per above 2 rules. # podman run -it --net=host quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:48d93321ca63acb5424cfaf5e603d398b08f5843ba542bb745139666fa52eda9 node-ip show By modifying /etc/systemd/system/kubelet.service.d/20-nodenet.conf file manually and then restarting node might not be helpful as upon restarting node nodeip-configuration service will also be restarted and /etc/systemd/system/kubelet.service.d/20-nodenet.conf file will be overridden. I have created a RFE to let user customize node IP in kubelet service but that has been Deferred as part of (or rather, a use case of) a larger multi-interface architecture that is being designed. https://issues.redhat.com/browse/RFE-1645 Hello Team, I tried the same by creating a new nodeconfig file(with higher number like 98-node-ip.conf), but in my customer's environment, there isn't any "--node-ip" parameter line in kubelet.service file so it is not getting the new defined IP in nodeconfig. If I forcefully update the kubelet.config and reload the daemon, it is working but whenever the node reboots, the newly added line got removed as the default machine-config worker render doesn't have that parameter. Also, kubelet.service updation through machine-config is not recommended and not supported. Customer had 4.5 version earlier, but now they have upgraded to 4.6.31 version. I could see in 4.5 version this parameter wasn't there but introduced in 4.6+ version. But even after upgrade, they don't have --node-ip in kubelet.service file. The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days |