[RCA] unable to ensure pod container exists: failed to create container for /kubepods/burstable/podd6089668-b695-11e8-9873-02c892353296 : dbus: connection closed by user
Description of problem:
The error we get is the following :
unable to ensure pod container exists: failed to create container for /kubepods/burstable/podd6089668-b695-11e8-9873-02c892353296 : dbus: connection closed by user
Example:
oc describe pod eaa-pa-cmdb-sit-1-hd86c
.
.
.
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 28s default-scheduler Successfully assigned eaa-pa-cmdb-sit-1-hd86c to ip-172-28-248-145.eu-central-1.compute.internal
Normal SuccessfulMountVolume 28s kubelet, ip-172-28-248-145.eu-central-1.compute.internal MountVolume.SetUp succeeded for volume "default-token-twxfj"
Normal SuccessfulMountVolume 28s kubelet, ip-172-28-248-145.eu-central-1.compute.internal MountVolume.SetUp succeeded for volume "preprod-eaa-sit-volume"
Warning FailedCreatePodContainer 3s (x3 over 28s) kubelet, ip-172-28-248-145.eu-central-1.compute.internal unable to ensure pod container exists: failed to create container for /kubepods/burstable/podd6089668-b695-11e8-9873-02c892353296 : dbus: connection closed by user
Problem solved with:
systemctl restart systemd-logind
systemctl restart dbus
systemctl restart dnsmasq NetworkManager
systemctl restart atomic-openshift-node.service
We need assistance to find the root cause:
Attach the sosreport of the affected node on fubar:
fubar.gsslab.rdu2.redhat.com:/fubar/02185872
Found some interesting logs on journalctl_--no-pager_--unit_atomic-openshift-node that may help to find the RCA.
/journalctl_--no-pager_--unit_atomic-openshift-node:Sep 05 15:42:57 ip-172-28-248-64.eu-central-1.compute.internal atomic-openshift-node[23289]: E0905 15:42:57.738883 23289 dnsmasq.go:105] unable to periodically refresh dnsmasq status: dbus: connection closed by user
./journalctl_--no-pager_--unit_atomic-openshift-node:Sep 05 15:43:27 ip-172-28-248-64.eu-central-1.compute.internal atomic-openshift-node[23289]: E0905 15:43:27.744564 23289 dnsmasq.go:105] unable to periodically refresh dnsmasq status: dbus: connection closed by user
./journalctl_--no-pager_--unit_atomic-openshift-node:Sep 05 15:43:57 ip-172-28-248-64.eu-central-1.compute.internal atomic-openshift-node[23289]: E0905 15:43:57.750316 23289 dnsmasq.go:105] unable to periodically refresh dnsmasq status: dbus: connection closed by user
./journalctl_--no-pager_--unit_atomic-openshift-node:Sep 05 15:44:27 ip-172-28-248-64.eu-central-1.compute.internal atomic-openshift-node[23289]: E0905 15:44:27.750543 23289 dnsmasq.go:105] unable to periodically refresh dnsmasq status: dbus: connection closed by user
./journalctl_--no-pager_--unit_atomic-openshift-node:Sep 05 15:44:57 ip-172-28-248-64.eu-central-1.compute.internal atomic-openshift-node[23289]: E0905 15:44:57.750769 23289 dnsmasq.go:105] unable to periodically refresh dnsmasq status: dbus: connection closed by user
./journalctl_--no-pager_--unit_atomic-openshift-node:Sep 05 15:45:27 ip-172-28-248-64.eu-central-1.compute.internal atomic-openshift-node[23289]: E0905 15:45:27.751035 23289 dnsmasq.go:105] unable to periodically refresh dnsmasq status: dbus: connection closed by user
./journalctl_--no-pager_--unit_atomic-openshift-node:Sep 05 15:45:57 ip-172-28-248-64.eu-central-1.compute.internal atomic-openshift-node[23289]: E0905 15:45:57.751272 23289 dnsmasq.go:105] unable to periodically refresh dnsmasq status: dbus: connection closed by user
Version-Release number of selected component (if applicable):
OCP 3.9
OCP 3.6-3.10 is no longer on full support [1]. Marking CLOSED DEFERRED. If you have a customer case with a support exception or have reproduced on 3.11+, please reopen and include those details. When reopening, please set the Target Release to the appropriate version where needed.
[1]: https://access.redhat.com/support/policy/updates/openshift