Bug 1817594 - nodeip-configuration 'Failed to find suitable node ip'
Summary: nodeip-configuration 'Failed to find suitable node ip'
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Machine Config Operator
Version: 4.4
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 4.4.0
Assignee: Antoni Segura Puimedon
QA Contact: Victor Voronkov
URL:
Whiteboard:
Depends On: 1819484
Blocks: 1771572
TreeView+ depends on / blocked
 
Reported: 2020-03-26 15:58 UTC by Eduardo Minguez
Modified: 2020-05-31 13:33 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1819484 (view as bug list)
Environment:
Last Closed: 2020-05-04 11:47:28 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github openshift machine-config-operator issues 1585 None open nodeip-configuration 'Failed to find suitable node ip' 2020-06-30 07:31:39 UTC
Github openshift machine-config-operator pull 1616 None closed Bug 1817594: [release-4.4] Nodeip retry on failure 2020-06-30 07:31:39 UTC
Red Hat Product Errata RHBA-2020:0581 None None None 2020-05-04 11:48:13 UTC

Description Eduardo Minguez 2020-03-26 15:58:50 UTC
Description of problem:
In recent OCP4 versions, nodeip-configuration service is failing:

[root@kni1-worker-0 ~]# systemctl status nodeip-configuration.service
● nodeip-configuration.service - Writes IP address configuration so that kubelet and crio services select a valid node IP
   Loaded: loaded (/etc/systemd/system/nodeip-configuration.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2020-03-25 13:47:46 UTC; 1h 10min ago
  Process: 2092 ExecStart=/usr/local/bin/nodeip-finder 10.19.138.14 (code=exited, status=1/FAILURE)
 Main PID: 2092 (code=exited, status=1/FAILURE)
      CPU: 106ms
 
Mar 25 13:47:46 localhost.localdomain systemd[1]: Starting Writes IP address configuration so that kubelet and crio services select a valid node IP...
Mar 25 13:47:46 localhost.localdomain nodeip-finder[2092]: Filtering out Address(127.0.0.1/8, dev=lo) due to it having host scope
Mar 25 13:47:46 localhost.localdomain nodeip-finder[2092]: Filtering out Address(::1/128, dev=lo) due to it having host scope
Mar 25 13:47:46 localhost.localdomain nodeip-finder[2092]: Is 10.19.138.14 between 172.22.0.0 and 172.22.0.255
Mar 25 13:47:46 localhost.localdomain nodeip-finder[2092]: Is 10.19.138.14 between fe80:: and fe80::ffff:ffff:ffff:ffff
Mar 25 13:47:46 localhost.localdomain nodeip-finder[2092]: Failed to find suitable node ip
Mar 25 13:47:46 localhost.localdomain systemd[1]: nodeip-configuration.service: Main process exited, code=exited, status=1/FAILURE
Mar 25 13:47:46 localhost.localdomain systemd[1]: nodeip-configuration.service: Failed with result 'exit-code'.
Mar 25 13:47:46 localhost.localdomain systemd[1]: Failed to start Writes IP address configuration so that kubelet and crio services select a valid node IP.
Mar 25 13:47:46 localhost.localdomain systemd[1]: nodeip-configuration.service: Consumed 106ms CPU time

It seems the problem is the service was executed before an address in the vip subnet was set.

Version-Release number of selected component (if applicable):
4.4.0-202003230106 at least (4.4 RC0)

How reproducible:
It seems a race condition because sometimes it works.

Steps to Reproduce:
1. Deploy an IPI baremetal cluster
2. 
3.

Actual results:
Installation fails due to timeout.

Expected results:
Installation finish successfully.

Additional info:
The service has After=network-online.target dependency but according to the docs:

Units that strictly require a configured network connection should pull in network-online.target (via a Wants= type dependency) and order themselves after it. This target unit is intended to pull in a service that delays further execution until the network is sufficiently set up. What precisely this requires is left to the implementation of the network managing service.

So I'm not sure if NetworkManager starts the network-online.target before having that IP and if that's the case, maybe an ExecStartPre directive to ping the gateway or something in that direction will fix it.

Comment 1 Antoni Segura Puimedon 2020-03-30 12:55:46 UTC
The issue is:

* NetworkManager-wait-online will only wait for whichever networking comes up first (which is not necessarily the control plane network we need) or 30 seconds (whichever comes first).
* The nodeip configuration service runs as a oneshot systemd service and if it fails it does not restart.

We'll either leverage newer systemd support for retry on failure oneshot services or add the mechanism to the executables being called by the service.

Comment 2 Antoni Segura Puimedon 2020-03-30 13:15:56 UTC
The commit to systemd that adds the restart capability is:


    10e72727ee - (6 months ago) Allow restart for oneshot units — Claudio Zumbo

This got included to systemd releases:

    $ git tag --contains 10e72727ee
    v244
    v244-rc1
    v245
    v245-rc1
    v245-rc2

So most likely we need to implement the retry mechanism in our executables.

Comment 5 Dana Safford 2020-04-02 18:54:34 UTC
As this is becoming important, I raised the Customer Escalation Flag.

Comment 6 Steve Milner 2020-04-03 13:15:13 UTC
Updating to 4.4.0 target per https://github.com/openshift/machine-config-operator/pull/1601#issuecomment-608425178

Comment 9 Victor Voronkov 2020-04-10 09:15:46 UTC
Verified on 4.4.0-0.ci-2020-04-09-133825

we modified the systemd configuration to let the nodeip service start before network-online.target

sudo vi /etc/systemd/system/nodeip-configuration.service
[Unit]
Description=Writes IP address configuration so that kubelet and crio services select a valid node IP
# This only applies to VIP managing environments where the kubelet and crio IP
# address picking logic is flawed and may end up selecting an address from a
# different subnet or a deprecated address
#Wants=network-online.target
After=ignition-firstboot-complete.service
Before=kubelet.service crio.service

[Service]
# Need oneshot to delay kubelet
Type=oneshot
ExecStart=/usr/local/bin/nodeip-finder --retry-on-failure fd2e:6f44:5dd8::5

[Install]
WantedBy=multi-user.target



##### then rebooted the node and watch the log:

journalctl -u nodeip-configuration.service
-- Reboot --
Apr 10 09:03:54 localhost systemd[1]: Starting Writes IP address configuration so that kubelet and crio services select a valid node IP...
Apr 10 09:03:55 master-0-0 nodeip-finder[1362]: Filtering out Address(127.0.0.1/8, dev=lo) due to it having host scope
Apr 10 09:03:55 master-0-0 nodeip-finder[1362]: Filtering out Address(::1/128, dev=lo) due to it having host scope
Apr 10 09:03:55 master-0-0 nodeip-finder[1362]: Failed to find suitable node ip. Retrying...
Apr 10 09:03:55 master-0-0 nodeip-finder[1362]: Filtering out Address(127.0.0.1/8, dev=lo) due to it having host scope
Apr 10 09:03:55 master-0-0 nodeip-finder[1362]: Filtering out Address(::1/128, dev=lo) due to it having host scope
Apr 10 09:03:55 master-0-0 nodeip-finder[1362]: Is 192.168.123.5 between fe80:: and fe80::ffff:ffff:ffff:ffff
Apr 10 09:03:55 master-0-0 nodeip-finder[1362]: Is 192.168.123.5 between 192.168.123.0 and 192.168.123.255
Apr 10 09:03:55 master-0-0 nodeip-finder[1362]: Is 192.168.123.5 between fe80:: and fe80::ffff:ffff:ffff:ffff
Apr 10 09:03:55 master-0-0 nodeip-finder[1362]: VIP Subnet 192.168.123.0/24
Apr 10 09:03:55 master-0-0 nodeip-finder[1362]: Processing CustomAction for target
Apr 10 09:03:55 master-0-0 nodeip-finder[1362]:   parser = 140016845542176
Apr 10 09:03:55 master-0-0 nodeip-finder[1362]:   values = '192.168.123.5'
Apr 10 09:03:55 master-0-0 nodeip-finder[1362]:   option_string = None
Apr 10 09:03:55 master-0-0 systemd[1]: Started Writes IP address configuration so that kubelet and crio services select a valid node IP.
Apr 10 09:03:55 master-0-0 systemd[1]: nodeip-configuration.service: Consumed 162ms CPU time

[core@master-0-0 ~]$ sudo systemctl status crio.service
Warning: The unit file, source configuration file or drop-ins of crio.service changed on disk. Run 'systemctl daemon-reload' to reload units.
● crio.service - Open Container Initiative Daemon
   Loaded: loaded (/usr/lib/systemd/system/crio.service; disabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/crio.service.d
           └─10-default-env.conf, 20-nodenet.conf, 20-stream-address.conf
   Active: active (running) since Fri 2020-04-10 09:04:26 UTC; 5min ago
     Docs: https://github.com/cri-o/cri-o
 Main PID: 1553 (crio)
    Tasks: 50
   Memory: 186.1M
      CPU: 53.911s
   CGroup: /system.slice/crio.service
           ├─ 1553 /usr/bin/crio --stream-address=192.168.123.132 --enable-metrics=true --metrics-port=9537
           └─29701 /usr/libexec/crio/conmon -c bb1020474ebec33f1322f277591a265dd80cc0e8be2173e65c9d0944b0f635e6 -n k8s_etcd_etcd-master-0-0_openshift-etc

cluster state is good, all nodes ready

Comment 11 errata-xmlrpc 2020-05-04 11:47:28 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581


Note You need to log in before you can comment on or make changes to this bug.