Description of problem: IHAC where the nodes have each 3 interfaces. The default route point to interface 0 where the OpenShift should listen on interface 1. In some cases, when the system is rebooted, the kubelet and all the services (including on master) are bound to interface0 where they should bound to interface1. The problem causes that ETCD member can't join the cluster due to the incorrect IP address. Once the interface0 is disabled, the kubelet and other services are bound to interface1. How to force the kubelet to bound to correct interface? In the list - "ip addr sh", the interface1 is before the interface0. Version-Release number of selected component (if applicable): OpenShift Container Platform 4.5 network plugin - openshift-sdn How reproducible: - cluster with 2 nics and more - default route points to different nic Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Could you please provide an update on this ?
The Kubelet's node IP can be pinned to an IP by creating a file like this, and then rebooting afterwards: cat <<EOF > /etc/systemd/system/kubelet.service.d/20-nodenet.conf [Service] Environment="KUBELET_NODE_IP=10.0.1.2" EOF You will need to reboot the node after creating the file.
(In reply to Ryan Phillips from comment #4) > The Kubelet's node IP can be pinned to an IP by creating a file like this, > and then rebooting afterwards: > > cat <<EOF > /etc/systemd/system/kubelet.service.d/20-nodenet.conf > [Service] > Environment="KUBELET_NODE_IP=10.0.1.2" > EOF > > You will need to reboot the node after creating the file. Great, I will KCS that.
Hello Team, nodeip-configuration service is responsible to set KUBELET_NODE_IP in /etc/systemd/system/kubelet.service.d/20-nodenet.conf file. The kubelet.service reads KUBELET_NODE_IP from /etc/systemd/system/kubelet.service.d/20-nodenet.conf file. As part of nodeip-configuration service one container gets created and a script gets executed within the container to extract node IP. ~~~ ExecStart=/bin/bash -c " \ until \ /usr/bin/podman run --rm \ --authfile /var/lib/kubelet/config.json \ --net=host \ --volume /etc/systemd/system:/etc/systemd/system:z \ quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c0e6c751d3f42c280624f88bc91e1e27fa602a09fa57a41d0c34095d853e9e98 \ node-ip \ set --retry-on-failure; \ do \ sleep 5; \ done" ~~~ node IP is determined as per following rules. 1. the NIC is configured as the default gateway.[0] 2. the IP of the NIC is resolved as the node hostname. when 1 and 2 are satisfied, then that NIC IP would be set as the Openshift node IP. [0] [https://github.com/openshift/sdn/blob/7106dab9c5785ae163237d443de0ddc19cb1f040/vendor/k8s.io/apimachinery/pkg/util/net/interface.go#L395-L398] ~~~ // chooseHostInterfaceFromRoute cycles through each default route provided, looking for a // global IP address from the interface for the route. addressFamilies determines whether it // prefers IPv4 or IPv6 func chooseHostInterfaceFromRoute(routes []Route, nw networkInterfacer, addressFamilies AddressFamilyPreference) ~~~ If you run below command on any node, you will always see the node IP is chosen as per above 2 rules. # podman run -it --net=host quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:48d93321ca63acb5424cfaf5e603d398b08f5843ba542bb745139666fa52eda9 node-ip show By modifying /etc/systemd/system/kubelet.service.d/20-nodenet.conf file manually and then restarting node might not be helpful as upon restarting node nodeip-configuration service will also be restarted and /etc/systemd/system/kubelet.service.d/20-nodenet.conf file will be overridden. I have created a RFE to let user customize node IP in kubelet service but that has been Deferred as part of (or rather, a use case of) a larger multi-interface architecture that is being designed. https://issues.redhat.com/browse/RFE-1645
Hello Team, I tried the same by creating a new nodeconfig file(with higher number like 98-node-ip.conf), but in my customer's environment, there isn't any "--node-ip" parameter line in kubelet.service file so it is not getting the new defined IP in nodeconfig. If I forcefully update the kubelet.config and reload the daemon, it is working but whenever the node reboots, the newly added line got removed as the default machine-config worker render doesn't have that parameter. Also, kubelet.service updation through machine-config is not recommended and not supported. Customer had 4.5 version earlier, but now they have upgraded to 4.6.31 version. I could see in 4.5 version this parameter wasn't there but introduced in 4.6+ version. But even after upgrade, they don't have --node-ip in kubelet.service file.
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days