Bug 1904065

Summary: [release 4.6] [Openstack] HTTP_PROXY setting for NetworkManager-resolv-prepender not working
Product: OpenShift Container Platform Reporter: Emilien Macchi <emacchi>
Component: Machine Config OperatorAssignee: Emilien Macchi <emilien>
Machine Config Operator sub component: platform-openstack QA Contact: weiwei jiang <wjiang>
Status: CLOSED ERRATA Docs Contact:
Severity: urgent    
Priority: urgent CC: behoward, brad, eduen, emacchi, eslutsky, gcheresh, gparente, kgarriso, malonso, m.andre, mkrejci, mnguyen, openshift-bugs-escalate, pprinett, rcarrier, rheinzma, rioliu, tsohlber
Version: 4.6.zKeywords: UpcomingSprint
Target Milestone: ---   
Target Release: 4.6.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1873556
: 1907637 (view as bug list) Environment:
Last Closed: 2020-12-21 13:23:59 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1873556    
Bug Blocks: 1907636, 1907637    

Comment 1 Evgeny Slutsky 2020-12-09 13:26:10 UTC
this affects oVirt also

Comment 8 weiwei jiang 2020-12-18 06:08:02 UTC
Checked with 4.6.0-0.nightly-2020-12-17-202735, and it work well now.

$ oc get nodes -o wide 
NAME                                STATUS   ROLES    AGE    VERSION           INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                                                       KERNEL-VERSION                 CONTAINE
R-RUNTIME
wj46ios1218b-nbx79-master-0         Ready    master   105m   v1.19.0+7070803   192.168.1.73    <none>        Red Hat Enterprise Linux CoreOS 46.82.202012151054-0 (Ootpa)   4.18.0-193.37.1.el8_2.x86_64   cri-o://
1.19.0-26.rhaos4.6.git8a05a29.el8
wj46ios1218b-nbx79-master-1         Ready    master   105m   v1.19.0+7070803   192.168.1.179   <none>        Red Hat Enterprise Linux CoreOS 46.82.202012151054-0 (Ootpa)   4.18.0-193.37.1.el8_2.x86_64   cri-o://
1.19.0-26.rhaos4.6.git8a05a29.el8
wj46ios1218b-nbx79-master-2         Ready    master   105m   v1.19.0+7070803   192.168.0.86    <none>        Red Hat Enterprise Linux CoreOS 46.82.202012151054-0 (Ootpa)   4.18.0-193.37.1.el8_2.x86_64   cri-o://
1.19.0-26.rhaos4.6.git8a05a29.el8
wj46ios1218b-nbx79-worker-0-2mf7w   Ready    worker   91m    v1.19.0+7070803   192.168.3.184   <none>        Red Hat Enterprise Linux CoreOS 46.82.202012151054-0 (Ootpa)   4.18.0-193.37.1.el8_2.x86_64   cri-o://
1.19.0-26.rhaos4.6.git8a05a29.el8
wj46ios1218b-nbx79-worker-0-894rs   Ready    worker   92m    v1.19.0+7070803   192.168.1.62    <none>        Red Hat Enterprise Linux CoreOS 46.82.202012151054-0 (Ootpa)   4.18.0-193.37.1.el8_2.x86_64   cri-o://
1.19.0-26.rhaos4.6.git8a05a29.el8
wj46ios1218b-nbx79-worker-0-9zdhk   Ready    worker   19m    v1.19.0+7070803   192.168.2.159   <none>        Red Hat Enterprise Linux CoreOS 46.82.202012151054-0 (Ootpa)   4.18.0-193.37.1.el8_2.x86_64   cri-o://
1.19.0-26.rhaos4.6.git8a05a29.el8

$ oc get proxy cluster -o yaml
apiVersion: config.openshift.io/v1
kind: Proxy
metadata:
  creationTimestamp: "2020-12-18T04:14:43Z"
  generation: 1
  ......
  name: cluster
  resourceVersion: "347"
  selfLink: /apis/config.openshift.io/v1/proxies/cluster
  uid: 293f543f-41fa-4b26-88f4-8d2d9a6816ca
spec:
  httpProxy: http://proxy-user1:HIDDEN@10.0.77.163.nip.io:3128
  httpsProxy: http://proxy-user1:HIDDEN@10.0.77.163.nip.io:3128
  noProxy: rhos-d.infra.prod.upshift.rdu2.redhat.com,oauth-openshift.apps.wj46ios1218b.1218-39w.qe.rhcloud.com,api.wj46ios1218b.1218-39w.qe.rhcloud.com,api-int.wj46ios1218b.1218-39w.qe.rhcloud.com
  trustedCA:
    name: ""
status:
  httpProxy: http://proxy-user1:HIDDEN@10.0.77.163.nip.io:3128
  httpsProxy: http://proxy-user1:HIDDEN@10.0.77.163.nip.io:3128
  noProxy: .cluster.local,.svc,10.128.0.0/14,127.0.0.1,169.254.169.254,172.30.0.0/16,192.168.0.0/18,api-int.wj46ios1218b.1218-39w.qe.rhcloud.com,api.wj46ios1218b.1218-39w.qe.rhcloud.com,etcd-0.wj46ios1218b.1218-39w.qe.rhcloud.com,etcd-1.wj46ios1218b.1218-39w.qe.rhcloud.com,etcd-2.wj46ios1218b.1218-39w.qe.rhcloud.com,localhost,oauth-openshift.apps.wj46ios1218b.1218-39w.qe.rhcloud.com,rhos-d.infra.prod.upshift.rdu2.redhat.com

$ oc get nodes -o wide
NAME                                STATUS   ROLES    AGE    VERSION           INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                                                       KERNEL-VERSION                 CONTAINER-RUNTIME
wj46ios1218b-nbx79-master-0         Ready    master   106m   v1.19.0+7070803   192.168.1.73    <none>        Red Hat Enterprise Linux CoreOS 46.82.202012151054-0 (Ootpa)   4.18.0-193.37.1.el8_2.x86_64   cri-o://1.19.0-26.rhaos4.6.git8a05a29.el8
wj46ios1218b-nbx79-master-1         Ready    master   107m   v1.19.0+7070803   192.168.1.179   <none>        Red Hat Enterprise Linux CoreOS 46.82.202012151054-0 (Ootpa)   4.18.0-193.37.1.el8_2.x86_64   cri-o://1.19.0-26.rhaos4.6.git8a05a29.el8
wj46ios1218b-nbx79-master-2         Ready    master   107m   v1.19.0+7070803   192.168.0.86    <none>        Red Hat Enterprise Linux CoreOS 46.82.202012151054-0 (Ootpa)   4.18.0-193.37.1.el8_2.x86_64   cri-o://1.19.0-26.rhaos4.6.git8a05a29.el8
wj46ios1218b-nbx79-worker-0-2mf7w   Ready    worker   93m    v1.19.0+7070803   192.168.3.184   <none>        Red Hat Enterprise Linux CoreOS 46.82.202012151054-0 (Ootpa)   4.18.0-193.37.1.el8_2.x86_64   cri-o://1.19.0-26.rhaos4.6.git8a05a29.el8
wj46ios1218b-nbx79-worker-0-894rs   Ready    worker   94m    v1.19.0+7070803   192.168.1.62    <none>        Red Hat Enterprise Linux CoreOS 46.82.202012151054-0 (Ootpa)   4.18.0-193.37.1.el8_2.x86_64   cri-o://1.19.0-26.rhaos4.6.git8a05a29.el8
wj46ios1218b-nbx79-worker-0-9zdhk   Ready    worker   21m    v1.19.0+7070803   192.168.2.159   <none>        Red Hat Enterprise Linux CoreOS 46.82.202012151054-0 (Ootpa)   4.18.0-193.37.1.el8_2.x86_64   cri-o://1.19.0-26.rhaos4.6.git8a05a29.el8


$ oc debug nodes/wj46ios1218b-nbx79-master-0 -- chroot /host /usr/bin/cat /etc/NetworkManager/dispatcher.d/30-resolv-prepender                                                                               1 ↵
Creating debug namespace/openshift-debug-node-mlklp ...
Starting pod/wj46ios1218b-nbx79-master-0-debug ...
To use host binaries, run `chroot /host`
#!/bin/bash
set -eo pipefail
IFACE=$1
STATUS=$2

export HTTP_PROXY=http://proxy-user1:HIDDEN@10.0.77.163.nip.io:3128
export HTTPS_PROXY=http://proxy-user1:HIDDEN@10.0.77.163.nip.io:3128
export NO_PROXY=.cluster.local,.svc,10.128.0.0/14,127.0.0.1,169.254.169.254,172.30.0.0/16,192.168.0.0/18,api-int.wj46ios1218b.1218-39w.qe.rhcloud.com,api.wj46ios1218b.1218-39w.qe.rhcloud.com,etcd-0.wj46ios1218b.1218-39w.qe.rhcloud.com,etcd-1.wj46ios1218b.1218-39w.qe.rhcloud.com,etcd-2.wj46ios1218b.1218-39w.qe.rhcloud.com,localhost,oauth-openshift.apps.wj46ios1218b.1218-39w.qe.rhcloud.com,rhos-d.infra.prod.upshift.rdu2.redhat.com
case "$STATUS" in
    up|down|dhcp4-change|dhcp6-change)
    logger -s "NM resolv-prepender triggered by ${1} ${2}."

    # Ensure resolv.conf exists before we try to run podman
    if [[ ! -e /etc/resolv.conf ]] || ! grep -q nameserver /etc/resolv.conf; then
        cp /var/run/NetworkManager/resolv.conf /etc/resolv.conf
    fi

    NAMESERVER_IP=$(/usr/bin/podman run --rm \
        --authfile /var/lib/kubelet/config.json \
        --net=host \
        quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a34f5bdb0cd71e99b446fbc4598594f71b45b149df3506ff4a4d229605f76ed \
        node-ip \
        show \
        "192.168.0.5" \
        "192.168.0.7")
    DOMAIN="wj46ios1218b.1218-39w.qe.rhcloud.com"
    if [[ -n "$NAMESERVER_IP" ]]; then
        logger -s "NM resolv-prepender: Prepending 'nameserver $NAMESERVER_IP' to /etc/resolv.conf (other nameservers from /var/run/NetworkManager/resolv.conf)"
        sed -e "/^search/d" \
            -e "/Generated by/c# Generated by OpenStack resolv prepender NM dispatcher script\nsearch $DOMAIN\nnameserver $NAMESERVER_IP" \
            /var/run/NetworkManager/resolv.conf > /etc/resolv.tmp
    fi
    # Only leave the first 3 nameservers in /etc/resolv.conf
    sed -i ':a $!{N; ba}; s/\(^\|\n\)nameserver/\n# nameserver/4g' /etc/resolv.tmp
    mv -f /etc/resolv.tmp /etc/resolv.conf
    ;;
    *)
    ;;
esac

Removing debug pod ...
Removing debug namespace/openshift-debug-node-mlklp ...

Comment 10 errata-xmlrpc 2020-12-21 13:23:59 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.6.9 security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5614

Comment 11 Ben Nemec 2021-02-12 21:16:26 UTC
*** Bug 1884821 has been marked as a duplicate of this bug. ***