Bug 1873556 - [Openstack] HTTP_PROXY setting for NetworkManager-resolv-prepender not working
Summary: [Openstack] HTTP_PROXY setting for NetworkManager-resolv-prepender not working
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Machine Config Operator
Version: 4.5
Hardware: Unspecified
OS: Unspecified
medium
low
Target Milestone: ---
: 4.7.0
Assignee: Emilien Macchi
QA Contact: weiwei jiang
URL:
Whiteboard:
Depends On:
Blocks: 1904065 1907637
TreeView+ depends on / blocked
 
Reported: 2020-08-28 15:34 UTC by Robert Heinzmann
Modified: 2023-12-15 19:04 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: The proxy environment variables (HTTP_PROXY, HTTPS_PROXY, and NO_PROXY) were not loaded in the environment during the execution of the NetworkManager's resolv-prepender dispatcher. Consequence: Nodes fail to pull container images from remote registries when using a proxy. Fix: Export the HTTP_PROXY, HTTPS_PROXY and NO_PROXY variables. Result: Nodes can now pull the required container images when using a proxy.
Clone Of:
: 1904065 (view as bug list)
Environment:
Last Closed: 2021-02-24 15:16:47 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift machine-config-operator pull 2266 0 None closed Bug 1873556: [on-prem] inject the proxy into the env for NetworkManager.service 2021-02-17 18:31:53 UTC
Red Hat Product Errata RHSA-2020:5633 0 None None None 2021-02-24 15:17:27 UTC

Comment 2 Martin André 2020-08-28 18:41:43 UTC
Setting the priority to low as this shouldn't be triggered under normal circumstances, after https://bugzilla.redhat.com/show_bug.cgi?id=1870285.

As a workaround it is possible to use an IP address instead of a hostname for the proxy.

Comment 11 weiwei jiang 2020-12-09 07:53:19 UTC
Checked with 4.7.0-0.nightly-2020-12-04-013308, and now it should work well, so move to verified.

$ oc get proxy -A -o yaml                                                                                                                                                                                                                                               130 ↵
apiVersion: v1
items:
- apiVersion: config.openshift.io/v1
  kind: Proxy
  metadata:
    creationTimestamp: "2020-12-09T05:01:47Z"
    generation: 1
    ...
    name: cluster
    resourceVersion: "3039"
    selfLink: /apis/config.openshift.io/v1/proxies/cluster
    uid: 6237fddd-dba6-44c8-a67d-3a703824efbc
  spec:
    httpProxy: http://proxy-user1:HIDDEN@10.0.77.163.nip.io:3128
    httpsProxy: http://proxy-user1:HIDDEN@10.0.77.163.nip.io:3128
    noProxy: rhos-d.infra.prod.upshift.rdu2.redhat.com,oauth-openshift.apps.wj47ios1209c.1209-0eq.qe.rhcloud.com,api.wj47ios1209c.1209-0eq.qe.rhcloud.com,api-int.wj47ios1209c.1209-0eq.qe.rhcloud.com
    trustedCA:
      name: ""
  status:
    httpProxy: http://proxy-user1:HIDDEN@10.0.77.163.nip.io:3128
    httpsProxy: http://proxy-user1:HIDDEN@10.0.77.163.nip.io:3128
    noProxy: .cluster.local,.svc,10.128.0.0/14,127.0.0.1,169.254.169.254,172.30.0.0/16,192.168.0.0/18,api-int.wj47ios1209c.1209-0eq.qe.rhcloud.com,api.wj47ios1209c.1209-0eq.qe.rhcloud.com,etcd-0.,etcd-1.,etcd-2.,localhost,oauth-openshift.apps.wj47ios1209c.1209-0eq.qe.rhcloud.com,rhos-d.infra.prod.upshift.rdu2.redhat.com
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""


# Before scaleup: 
$ oc get machine -A                                                                                                                                                                                                                                                           
NAMESPACE               NAME                                PHASE     TYPE        REGION      ZONE   AGE                                                                                                                                                                        
openshift-machine-api   wj47ios1209c-clk5g-master-0         Running   m1.xlarge   regionOne   nova   109m                                                                                                                                                                       
openshift-machine-api   wj47ios1209c-clk5g-master-1         Running   m1.xlarge   regionOne   nova   109m                                                                                                                                                                       
openshift-machine-api   wj47ios1209c-clk5g-master-2         Running   m1.xlarge   regionOne   nova   109m                                                                                                                                                                       
openshift-machine-api   wj47ios1209c-clk5g-worker-0-rl4dp   Running   m1.large    regionOne   nova   102m               
openshift-machine-api   wj47ios1209c-clk5g-worker-0-t7bdk   Running   m1.large    regionOne   nova   102m
$ oc get nodes -o wide                                                                                                                                                                                                                                                        
NAME                                STATUS   ROLES    AGE    VERSION           INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                                                       KERNEL-VERSION                CONTAINER-RUNTIME
wj47ios1209c-clk5g-master-0         Ready    master   109m   v1.19.2+ad738ba   192.168.2.232   <none>        Red Hat Enterprise Linux CoreOS 47.83.202012032113-0 (Ootpa)   4.18.0-240.1.1.el8_3.x86_64   cri-o://1.20.0-0.rhaos4.7.git5e950f8.el8.26-dev
wj47ios1209c-clk5g-master-1         Ready    master   110m   v1.19.2+ad738ba   192.168.3.88    <none>        Red Hat Enterprise Linux CoreOS 47.83.202012032113-0 (Ootpa)   4.18.0-240.1.1.el8_3.x86_64   cri-o://1.20.0-0.rhaos4.7.git5e950f8.el8.26-dev
wj47ios1209c-clk5g-master-2         Ready    master   109m   v1.19.2+ad738ba   192.168.3.206   <none>        Red Hat Enterprise Linux CoreOS 47.83.202012032113-0 (Ootpa)   4.18.0-240.1.1.el8_3.x86_64   cri-o://1.20.0-0.rhaos4.7.git5e950f8.el8.26-dev
wj47ios1209c-clk5g-worker-0-rl4dp   Ready    worker   91m    v1.19.2+ad738ba   192.168.2.168   <none>        Red Hat Enterprise Linux CoreOS 47.83.202012032113-0 (Ootpa)   4.18.0-240.1.1.el8_3.x86_64   cri-o://1.20.0-0.rhaos4.7.git5e950f8.el8.26-dev                       
wj47ios1209c-clk5g-worker-0-t7bdk   Ready    worker   91m    v1.19.2+ad738ba   192.168.0.87    <none>        Red Hat Enterprise Linux CoreOS 47.83.202012032113-0 (Ootpa)   4.18.0-240.1.1.el8_3.x86_64   cri-o://1.20.0-0.rhaos4.7.git5e950f8.el8.26-dev



# After scaleup:
$ oc get machine -A
NAMESPACE               NAME                                PHASE     TYPE        REGION      ZONE   AGE
openshift-machine-api   wj47ios1209c-clk5g-master-0         Running   m1.xlarge   regionOne   nova   121m
openshift-machine-api   wj47ios1209c-clk5g-master-1         Running   m1.xlarge   regionOne   nova   121m
openshift-machine-api   wj47ios1209c-clk5g-master-2         Running   m1.xlarge   regionOne   nova   121m
openshift-machine-api   wj47ios1209c-clk5g-worker-0-p6p4q   Running   m1.large    regionOne   nova   4m9s
openshift-machine-api   wj47ios1209c-clk5g-worker-0-rl4dp   Running   m1.large    regionOne   nova   114m
openshift-machine-api   wj47ios1209c-clk5g-worker-0-t7bdk   Running   m1.large    regionOne   nova   114m
# openstack server list --name wj4 
+--------------------------------------+-----------------------------------+--------+--------------------------------------------+----------------------------+-----------+
| ID                                   | Name                              | Status | Networks                                   | Image                      | Flavor    |
+--------------------------------------+-----------------------------------+--------+--------------------------------------------+----------------------------+-----------+
| 4adb466b-b689-4b85-87de-fde162052331 | wj47ios1209c-clk5g-worker-0-rl4dp | ACTIVE | wj47ios1209c-clk5g-openshift=192.168.2.168 | rhcos-47.83.202012030221-0 | m1.large  |
| 0a704d97-2234-4f16-80da-b4501a8e2939 | wj47ios1209c-clk5g-worker-0-rdjl4 | ACTIVE | wj47ios1209c-clk5g-openshift=192.168.1.224 | rhcos-47.83.202012030221-0 | m1.large  |
| ffe86ac3-c19f-4eec-b243-da57fa18bd3d | wj47ios1209c-clk5g-worker-0-t7bdk | ACTIVE | wj47ios1209c-clk5g-openshift=192.168.0.87  | rhcos-47.83.202012030221-0 | m1.large  |
| e731c881-eed0-4537-b0ae-e30c972991ad | wj47ios1209c-clk5g-master-2       | ACTIVE | wj47ios1209c-clk5g-openshift=192.168.3.206 | rhcos-47.83.202012030221-0 | m1.xlarge |
| be467f71-fc41-4e85-975a-fcf075ccf599 | wj47ios1209c-clk5g-master-0       | ACTIVE | wj47ios1209c-clk5g-openshift=192.168.2.232 | rhcos-47.83.202012030221-0 | m1.xlarge |
| db3003fd-5c2f-4a6d-b8a6-d492dc13240b | wj47ios1209c-clk5g-master-1       | ACTIVE | wj47ios1209c-clk5g-openshift=192.168.3.88  | rhcos-47.83.202012030221-0 | m1.xlarge |
+--------------------------------------+-----------------------------------+--------+--------------------------------------------+----------------------------+-----------+
$ oc get nodes -o wide 
NAME                                STATUS   ROLES    AGE    VERSION           INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                                                       KERNEL-VERSION                CONTAINER-RUNTIME
wj47ios1209c-clk5g-master-0         Ready    master   138m   v1.19.2+ad738ba   192.168.2.232   <none>        Red Hat Enterprise Linux CoreOS 47.83.202012032113-0 (Ootpa)   4.18.0-240.1.1.el8_3.x86_64   cri-o://1.20.0-0.rhaos4.7.git5e950f8.el8.26-dev
wj47ios1209c-clk5g-master-1         Ready    master   138m   v1.19.2+ad738ba   192.168.3.88    <none>        Red Hat Enterprise Linux CoreOS 47.83.202012032113-0 (Ootpa)   4.18.0-240.1.1.el8_3.x86_64   cri-o://1.20.0-0.rhaos4.7.git5e950f8.el8.26-dev
wj47ios1209c-clk5g-master-2         Ready    master   138m   v1.19.2+ad738ba   192.168.3.206   <none>        Red Hat Enterprise Linux CoreOS 47.83.202012032113-0 (Ootpa)   4.18.0-240.1.1.el8_3.x86_64   cri-o://1.20.0-0.rhaos4.7.git5e950f8.el8.26-dev
wj47ios1209c-clk5g-worker-0-p6p4q   Ready    worker   19m    v1.19.2+ad738ba   192.168.2.18    <none>        Red Hat Enterprise Linux CoreOS 47.83.202012032113-0 (Ootpa)   4.18.0-240.1.1.el8_3.x86_64   cri-o://1.20.0-0.rhaos4.7.git5e950f8.el8.26-dev
wj47ios1209c-clk5g-worker-0-rl4dp   Ready    worker   120m   v1.19.2+ad738ba   192.168.2.168   <none>        Red Hat Enterprise Linux CoreOS 47.83.202012032113-0 (Ootpa)   4.18.0-240.1.1.el8_3.x86_64   cri-o://1.20.0-0.rhaos4.7.git5e950f8.el8.26-dev
wj47ios1209c-clk5g-worker-0-t7bdk   Ready    worker   120m   v1.19.2+ad738ba   192.168.0.87    <none>        Red Hat Enterprise Linux CoreOS 47.83.202012032113-0 (Ootpa)   4.18.0-240.1.1.el8_3.x86_64   cri-o://1.20.0-0.rhaos4.7.git5e950f8.el8.26-dev

[root@wj47ios1209c-clk5g-worker-0-p6p4q core]# cat /etc/NetworkManager/dispatcher.d/30-resolv-prepender 
#!/bin/bash
set -eo pipefail
IFACE=$1
STATUS=$2

export HTTP_PROXY=http://proxy-user1:HIDDEN@10.0.77.163.nip.io:3128
export HTTPS_PROXY=http://proxy-user1:HIDDEN@10.0.77.163.nip.io:3128
export NO_PROXY=.cluster.local,.svc,10.128.0.0/14,127.0.0.1,169.254.169.254,172.30.0.0/16,192.168.0.0/18,api-int.wj47ios1209c.1209-0eq.qe.rhcloud.com,api.wj47ios1209c.1209-0eq.qe.rhcloud.com,etcd-0.,etcd-1.,etcd-2.,localhost,oauth-openshift.apps.wj47ios1209c.1209-0eq.qe.rhcloud.com,rhos-d.infra.prod.upshift.rdu2.redhat.com

Comment 13 Yu Qi Zhang 2021-01-06 17:00:47 UTC
This should be documented as a bugfix. @Emilien could you add to the Doc Text field above?

Comment 15 errata-xmlrpc 2021-02-24 15:16:47 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633


Note You need to log in before you can comment on or make changes to this bug.