Bug 1851344 - nodeip-configuration.service service not starting on proxied installations
Summary: nodeip-configuration.service service not starting on proxied installations
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 4.6
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.6.0
Assignee: Martin André
QA Contact: David Sanz
URL:
Whiteboard:
Depends On:
Blocks: 1849036 1866379 1868178
TreeView+ depends on / blocked
 
Reported: 2020-06-26 09:26 UTC by David Sanz
Modified: 2020-10-27 16:10 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1866379 1868178 (view as bug list)
Environment:
Last Closed: 2020-10-27 16:09:46 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift machine-config-operator pull 1874 0 None closed Bug 1851344: Pass proxy settings nodeip-configuration service for baremetal, vsphere, openstack 2021-01-31 17:12:33 UTC
Red Hat Knowledge Base (Solution) 5312791 0 None None None 2020-08-11 22:51:50 UTC
Red Hat Product Errata RHBA-2020:4196 0 None None None 2020-10-27 16:10:13 UTC

Description David Sanz 2020-06-26 09:26:08 UTC
Description of problem:

Installing IPI on OSP with proxy, nodeip-configuration.service is not starting because podman is trying to download a container from quay.io using a direct connection instead of using the proxy:

[root@mrnd-13-46-px-nvlt8-master-0 etc]# journalctl -u nodeip-configuration.service
-- Logs begin at Fri 2020-06-26 09:00:21 UTC, end at Fri 2020-06-26 09:17:30 UTC. --
Jun 26 09:07:27 mrnd-13-46-px-nvlt8-master-0 systemd[1]: Starting Writes IP address configuration so that kubelet and crio services select a valid node IP...
Jun 26 09:07:28 mrnd-13-46-px-nvlt8-master-0 podman[1531]: Trying to pull quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e9163b67ae38a9d9be54acd42bc1e3496d46>
Jun 26 09:08:21 mrnd-13-46-px-nvlt8-master-0 systemd[1]: nodeip-configuration.service: Main process exited, code=killed, status=15/TERM
Jun 26 09:08:21 mrnd-13-46-px-nvlt8-master-0 systemd[1]: nodeip-configuration.service: Failed with result 'signal'.
Jun 26 09:08:21 mrnd-13-46-px-nvlt8-master-0 systemd[1]: Stopped Writes IP address configuration so that kubelet and crio services select a valid node IP.
Jun 26 09:08:21 mrnd-13-46-px-nvlt8-master-0 systemd[1]: nodeip-configuration.service: Consumed 266ms CPU time
-- Reboot --
Jun 26 09:08:42 mrnd-13-46-px-nvlt8-master-0 systemd[1]: Starting Writes IP address configuration so that kubelet and crio services select a valid node IP...
Jun 26 09:08:43 mrnd-13-46-px-nvlt8-master-0 podman[1253]: Trying to pull quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e9163b67ae38a9d9be54acd42bc1e3496d46>
Jun 26 09:09:43 mrnd-13-46-px-nvlt8-master-0 podman[1253]:   Get https://quay.io/v2/: dial tcp 172.18.124.15:443: i/o timeout
Jun 26 09:09:43 mrnd-13-46-px-nvlt8-master-0 podman[1253]: Error: unable to pull quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e9163b67ae38a9d9be54acd42bc1e>
Jun 26 09:09:43 mrnd-13-46-px-nvlt8-master-0 systemd[1]: nodeip-configuration.service: Main process exited, code=exited, status=125/n/a
Jun 26 09:09:43 mrnd-13-46-px-nvlt8-master-0 systemd[1]: nodeip-configuration.service: Failed with result 'exit-code'.
Jun 26 09:09:43 mrnd-13-46-px-nvlt8-master-0 systemd[1]: Failed to start Writes IP address configuration so that kubelet and crio services select a valid node IP.
Jun 26 09:09:43 mrnd-13-46-px-nvlt8-master-0 systemd[1]: nodeip-configuration.service: Consumed 252ms CPU time


Note: quay.io resolves to 172.18.124.15 because it is the way we can block direct connections to internet.

Version-Release number of selected component (if applicable):
4.6.0-0.nightly-2020-06-26-035408

How reproducible:

Steps to Reproduce:
1.Install using proxy on install-config.yaml
proxy:
  httpProxy:   http://10.0.79.123:3128
  httpsProxy:  http://10.0.79.123:3128
  noProxy: 'rhos-d.infra.prod.upshift.rdu2.redhat.com,oauth-openshift.apps.mrnd-13-46-px.qe.devcluster.openshift.com'
2.Set DNS to block resolution on quay.io to catch any direct connection done by the node
3.Check status of nodeip-configuration.service

Actual results:
Service fails to start, it launch a container trying to download it directly from quay.io without using the proxy configured on the system

Expected results:
Service used the proxy configured on the system for any http connection

Additional info:

Comment 1 Martin André 2020-06-26 11:25:17 UTC
The nodeip-configuration.service unit file doesn't export the environment variables to use the proxy. 
I'll submit a tentative patch shortly

Comment 4 David Sanz 2020-07-10 10:21:07 UTC
Verified on 4.6.0-0.nightly-2020-07-10-092830

Comment 6 errata-xmlrpc 2020-10-27 16:09:46 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196


Note You need to log in before you can comment on or make changes to this bug.