Bug 1839900 - Racing condition with kubelet and hostname
Summary: Racing condition with kubelet and hostname
Keywords:
Status: CLOSED DUPLICATE of bug 1803962
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: RHCOS
Version: 4.3.z
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: ---
: ---
Assignee: Micah Abbott
QA Contact: Michael Nguyen
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-05-25 22:06 UTC by Odilon Sousa
Modified: 2023-09-07 23:12 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-05-26 13:21:47 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Odilon Sousa 2020-05-25 22:06:37 UTC
Description of problem:

After one outage on the hypervisor, the node was rebooted, and the kubelet could not start due to the hostname being localhost instead of the actual hostname of the node.

Version-Release number of selected component (if applicable):

OpenShift Container Platform 4.3.18

How reproducible:

Shutdown the node turn and start right after, sometime the kubelet will start without the right hostname.

Steps to Reproduce:
1. Reboot the node and check if the hostname is localhost
2. Kubelet will not work as expected
3. Set the right hostname
4. Restart the kubelet.

Actual results:

18 13:54:10 localhost crio[1340]: time="2020-05-18 13:54:10.358249819Z" level=error msg="CNI network \"\" not found"
May 18 13:54:10 localhost systemd[1]: Started Open Container Initiative Daemon.
May 18 13:54:10 localhost systemd[1]: Starting Kubernetes Kubelet...
May 18 13:54:11 localhost hyperkube[1895]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 18 13:54:11 localhost hyperkube[1895]: I0518 13:54:11.204217    1895 flags.go:33] FLAG: --add-dir-header="false"
May 18 13:54:11 localhost hyperkube[1895]: I0518 13:54:11.204308    1895 flags.go:33] FLAG: --address="0.0.0.0"
May 18 13:54:11 localhost hyperkube[1895]: I0518 13:54:11.204314    1895 flags.go:33] FLAG: --allowed-unsafe-sysctls="[]"
May 18 13:54:11 localhost hyperkube[1895]: I0518 13:54:11.204319    1895 flags.go:33] FLAG: --alsologtostderr="false"
May 18 13:54:11 localhost hyperkube[1895]: I0518 13:54:11.204322    1895 flags.go:33] FLAG: --anonymous-auth="true"

Expected results:

May 18 13:54:13 localhost hyperkube[1895]: E0518 13:54:13.301465    1895 kubelet.go:2278] node "localhost" not found
May 18 13:54:13 localhost NetworkManager[1211]: <info>  [1589810053.3015] policy: set-hostname: set hostname to 'server3.example.com' (from address lookup)
May 18 13:54:13 server3.example.com systemd-hostnamed[1243]: Changed host name to 'server3.example.com'


Additional info:

The node is running on VMware.

Comment 2 Odilon Sousa 2020-05-25 22:09:51 UTC
I don't know if this could be related to the https://bugzilla.redhat.com/show_bug.cgi?id=1803962 .

Comment 4 Colin Walters 2020-05-26 13:21:47 UTC

*** This bug has been marked as a duplicate of bug 1803962 ***

Comment 6 Colin Walters 2020-05-26 19:36:09 UTC
This should be fixed as of e.g. 4.3.22:

```
$ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.22-x86_64 | grep machine-config
  machine-config-operator                       https://github.com/openshift/machine-config-operator                       c6a1e9b3d022671cef735d55eb277c140556b301
$ cd ~/src/machine-config-operator
$ git shortlog --no-merges c6a1e9b3d022671cef735d55eb277c140556b301 --grep=network-online
Ryan Phillips (1):
      Bug 1763700: kubelet: add dependency on network-online.target

W. Trevor King (1):
      templates/_base/master/units/etcd-member: Block on network-online.target
```

IOW please check that

```
[root@api ~]# grep network-online /etc/systemd/system/kubelet.service
Wants=rpc-statd.service network-online.target crio.service
After=network-online.target crio.service
[root@api ~]# 
```


Note You need to log in before you can comment on or make changes to this bug.