Description of problem: When an installation is configured to use Proxy some Pods that require access to the OpenStack API, like openstack-cinder-csi-driver-operator or cluster-network-operator (when using Kuryr) or machine Pods attempt to connect to the API directly without using the Proxy causing installation to fail. oc logs -f machine-api-controllers-5689ccb7c4-jknxd -n openshift-machine-api -c machine-controller E0727 18:00:00.317614 1 controller.go:302] controller-runtime/manager/controller/machine_controller "msg"="Reconciler error" "error"="Error checking if instance exists (machine/actuator.go 346): \nError getting a new instance service from the machine (machine/actuator.go 467): Failed to authenticate provider client: Get \"https://38.x.x.91:13000/\": dial tcp 38.x.x.91:13000: connect: no route to host" "name"="ocp-central-48vt6-master-2" "namespace"="openshift-machine-api" Install-config.yaml used: apiVersion: v1 baseDomain: ci.vexxhost.cz compute: - name: worker platform: openstack: type: m1.xlarge additionalSecurityGroupIDs: ['b97c865e-95fa-4a92-8930-241425d33fd4'] replicas: 3 controlPlane: name: master platform: openstack: type: m1.xlarge additionalSecurityGroupIDs: ['b97c865e-95fa-4a92-8930-241425d33fd4'] replicas: 3 metadata: name: ocp-central networking: machineNetwork: - cidr: 172.16.0.0/24 platform: openstack: cloud: openshift machinesSubnet: 6bb82a4f-de17-4872-8898-94cafa8ac81d apiVIP: 172.16.0.5 ingressVIP: 172.16.0.7 defaultMachinePlatform: type: m1.xlarge proxy: httpProxy: http://dummy:dummy@172.16.0.61:3128/ httpsProxy: https://dummy:dummy@172.16.0.61:3130/ pullSecret: | sshKey: | additionalTrustBundle: <cloud-ca> <ca-configured-on-squid> $ openstack server list | d5e24ad5-d8e5-436a-8ac2-8f52651e7c9f | ocp-central-hnprb-master-2 | ACTIVE | proxy=172.16.0.126 | ocp-central-hnprb-rhcos | m1.xlarge | | d1945d8d-1cf8-4afd-8ff5-c427869467bc | ocp-central-hnprb-master-1 | ACTIVE | proxy=172.16.0.146 | ocp-central-hnprb-rhcos | m1.xlarge | | 02f16d66-b589-4e9e-ae97-3b6cf980cfac | ocp-central-hnprb-master-0 | ACTIVE | proxy=172.16.0.201 | ocp-central-hnprb-rhcos | m1.xlarge | | 7589d476-2d1c-4ad1-bdcd-1ff1db1e9282 | bastion-proxy | ACTIVE | installer-network=10.196.2.27, 38.x.x.131; proxy=172.16.0.61 | centos8-stream | m1.medium | +--------------------------------------+--------------------------------+--------+-----------------------------------------------------------------+-------------------------+-----------+ $ openstack router show bastion +-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | admin_state_up | UP | | availability_zone_hints | | | availability_zones | central | | created_at | 2021-07-21T19:38:00Z | | description | | | external_gateway_info | {"network_id": "7ca1777f-24ab-41cf-add1-e4c1d8b81725", "external_fixed_ips": [{"subnet_id": "29065cbb-a0f3-480c-998e-c5bbb3854656", "ip_address": "38.x.x.218"}], "enable_snat": true} | | flavor_id | None | | id | 52299708-5de4-4681-bda8-c60c89520632 | | interfaces_info | [{"port_id": "c169e0b2-84ca-4d79-b805-bb3dbbb36bc8", "ip_address": "10.196.0.1", "subnet_id": "112bc049-b03e-4dae-a8bf-ced6f9674ebd"}] Version: Squid configuration on bastion VM: [centos@bastion-proxy ~]$ sudo cat /etc/squid/squid.conf acl localnet src 0.0.0.0/0 acl SSL_ports port 443 acl SSL_ports port 53 acl SSL_ports port 1025-65535 acl Safe_ports port 80 acl Safe_ports port 53 acl Safe_ports port 443 acl Safe_ports port 1025-65535 acl CONNECT method CONNECT http_access deny !Safe_ports http_access deny CONNECT !SSL_ports http_access allow localnet http_access deny all http_port 3128 https_port 3130 cert=/etc/squid/certs/domain.crt key=/etc/squid/certs/domain.key cafile=/etc/squid/certs/domain.crt # Leave coredumps in the first cache dir coredump_dir /var/spool/squid auth_param basic program /usr/lib64/squid/basic_ncsa_auth /etc/squid/passwd auth_param basic children 5 auth_param basic realm Squid Basic Authentication auth_param basic credentialsttl 2 hours acl auth_users proxy_auth REQUIRED http_access allow auth_users $ openshift-install version 4.9.0-0.nightly-2021-07-25-025749 with IPI Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Verified on 4.9.0-0.nightly-2021-08-31-123131 on top of OSP 16.1.6 (RHOS-16.1-RHEL-8-20210604.n.0) All the operators are available after IPI installation on a restricted network using proxy, and machine-controller container is not trying to connect to the OSP API outside the proxy (the error message in the description of the BZ is not being shown). $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.9.0-0.nightly-2021-08-31-123131 True False 43h Cluster version is 4.9.0-0.nightly-2021-08-31-123131 $ oc get proxy cluster -o json | jq .status { "httpProxy": "http://dummy:dummy@172.16.0.3:3128/", "httpsProxy": "https://dummy:dummy@172.16.0.3:3130/", "noProxy": ".cluster.local,.svc,10.128.0.0/14,127.0.0.1,169.254.169.254,172.16.0.0/24,172.30.0.0/16,api-int.ostest.shiftstack.com,localhost" } $ oc -n openshift-machine-api get pods NAME READY STATUS RESTARTS AGE cluster-autoscaler-operator-76cf8dc889-jfjrj 2/2 Running 0 43h cluster-baremetal-operator-56fd57cd8d-gzpbt 2/2 Running 0 43h machine-api-controllers-5865dc5b55-z4d7q 7/7 Running 0 43h machine-api-operator-594ddf49fb-wd74j 2/2 Running 0 43h $ oc -n openshift-machine-api -c machine-controller rsh machine-api-controllers-5865dc5b55-z4d7q sh-4.4$ env | grep -i proxy HTTP_PROXY=http://dummy:dummy@172.16.0.3:3128/ NO_PROXY=.cluster.local,.svc,10.128.0.0/14,127.0.0.1,169.254.169.254,172.16.0.0/24,172.30.0.0/16,api-int.ostest.shiftstack.com,localhost HTTPS_PROXY=https://dummy:dummy@172.16.0.3:3130/
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:3759