Bug 1985486

Summary: Cluster Proxy not used during installation on OSP with Kuryr
Product: OpenShift Container Platform Reporter: Maysa Macedo <mdemaced>
Component: NetworkingAssignee: Maysa Macedo <mdemaced>
Networking sub component: kuryr QA Contact: Jon Uriarte <juriarte>
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: unspecified CC: rlobillo
Version: 4.8   
Target Milestone: ---   
Target Release: 4.10.0   
Hardware: Unspecified   
OS: Unspecified   
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: The cluster network operator was not enforcing usage of Proxy to allow communication to OpenStack API when Kuryr is used in a restricted installation with Proxy. Consequence: When the cluster is configured with Kuryr the cluster network operator is unable to connect to the OpenStack API and cluster installation can not progress. Fix: Ensure the cluster network operator uses the configured Proxy for OpenStack API communication. Result: The cluster network operator can communicate to the Openstack API through the Proxy and installation succeeds.
Story Points: ---
Clone Of:
: 2014021 (view as bug list) Environment:
Last Closed: 2022-03-12 04:36:01 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 2014021    

Description Maysa Macedo 2021-07-23 17:02:48 UTC
Description of problem:

When an installation is configured to use Proxy some Pods that require access to the OpenStack API, like cluster-network-operator attempt to connect to the API directly without using the Proxy causing installation to fail.

Install-config.yaml used:

apiVersion: v1                                                                                                                                                                                
baseDomain: ci.vexxhost.cz                                                                                                                                                                    
- name: worker                                                                                                                                                                                
      type: m1.xlarge                                                                                                                                                                         
      additionalSecurityGroupIDs: ['b97c865e-95fa-4a92-8930-241425d33fd4']                                                                                                                    
  replicas: 3                                                                                                                                                                                 
  name: master                                                                                                                                                                                
      type: m1.xlarge                                                                                                                                                                         
      additionalSecurityGroupIDs: ['b97c865e-95fa-4a92-8930-241425d33fd4']                                                                                                                    
  replicas: 3
  name: ocp-central                                               
  - cidr:
    cloud:             openshift
    machinesSubnet:   6bb82a4f-de17-4872-8898-94cafa8ac81d
      type: m1.xlarge
  httpProxy: http://dummy:dummy@
  httpsProxy: https://dummy:dummy@
pullSecret: |
sshKey: |
additionalTrustBundle: <cloud-ca> <ca-configured-on-squid>

$ openstack server list

| d5e24ad5-d8e5-436a-8ac2-8f52651e7c9f | ocp-central-hnprb-master-2     | ACTIVE | proxy=                                              | ocp-central-hnprb-rhcos | m1.xlarge |
| d1945d8d-1cf8-4afd-8ff5-c427869467bc | ocp-central-hnprb-master-1     | ACTIVE | proxy=                                              | ocp-central-hnprb-rhcos | m1.xlarge |
| 02f16d66-b589-4e9e-ae97-3b6cf980cfac | ocp-central-hnprb-master-0     | ACTIVE | proxy=                                              | ocp-central-hnprb-rhcos | m1.xlarge |
| 7589d476-2d1c-4ad1-bdcd-1ff1db1e9282 | bastion-proxy                  | ACTIVE | installer-network=, 38.x.x.131; proxy= | centos8-stream          | m1.medium |

$ openstack router show bastion
| Field                   | Value                                                                                                                                                            
| admin_state_up          | UP                                                                                                                                                               
| availability_zone_hints |                                                                                                                                                                  
| availability_zones      | central                                                                                                                                                          
| created_at              | 2021-07-21T19:38:00Z                                                                                                                                             
| description             |                                                                                                                                                                  
| external_gateway_info   | {"network_id": "7ca1777f-24ab-41cf-add1-e4c1d8b81725", "external_fixed_ips": [{"subnet_id": "29065cbb-a0f3-480c-998e-c5bbb3854656", "ip_address": "38.x.x.218"}], "enable_snat": true} |
| flavor_id               | None                                                                                                                                                             
| id                      | 52299708-5de4-4681-bda8-c60c89520632                                                                                                                             
| interfaces_info         | [{"port_id": "c169e0b2-84ca-4d79-b805-bb3dbbb36bc8", "ip_address": "", "subnet_id": "112bc049-b03e-4dae-a8bf-ced6f9674ebd"}]

Squid configuration on bastion VM:

[centos@bastion-proxy ~]$ sudo cat /etc/squid/squid.conf                                                                                                                                     
acl localnet src
acl SSL_ports port 443
acl SSL_ports port 53
acl SSL_ports port 1025-65535
acl Safe_ports port 80
acl Safe_ports port 53
acl Safe_ports port 443
acl Safe_ports port 1025-65535
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localnet
http_access deny all
http_port 3128
https_port 3130 cert=/etc/squid/certs/domain.crt key=/etc/squid/certs/domain.key cafile=/etc/squid/certs/domain.crt                                                                          

# Leave coredumps in the first cache dir
coredump_dir /var/spool/squid

auth_param basic program /usr/lib64/squid/basic_ncsa_auth /etc/squid/passwd
auth_param basic children 5
auth_param basic realm Squid Basic Authentication
auth_param basic credentialsttl 2 hours
acl auth_users proxy_auth REQUIRED
http_access allow auth_users

$ openshift-install version
4.8.0-0.nightly-2021-07-19-192457 with IPI

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

Actual results:

Expected results:

Additional info:

Comment 3 rlobillo 2021-10-05 14:18:12 UTC
Verified on 4.10.0-0.nightly-2021-10-04-213416 on top of RHOS-16.1-RHEL-8-20210818.n.0 with OVN-Octavia enabled.

The IPI installation of the cluster with NetworkType Kuryr on a restricted network connected to a proxy worked fine:

$ oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.10.0-0.nightly-2021-10-04-213416   True        False         68m     Cluster version is 4.10.0-0.nightly-2021-10-04-213416

$ oc get proxy cluster -o json
    "apiVersion": "config.openshift.io/v1",
    "kind": "Proxy",
    "metadata": {
        "creationTimestamp": "2021-10-05T11:40:12Z",
        "generation": 1,
        "name": "cluster",
        "resourceVersion": "399",
        "uid": "fd8cc9d5-536e-4f49-9a06-2298c9807583"
    "spec": {
        "httpProxy": "http://dummy:dummy@",
        "httpsProxy": "https://dummy:dummy@",
        "trustedCA": {
            "name": "user-ca-bundle"
    "status": {
        "httpProxy": "http://dummy:dummy@",
        "httpsProxy": "https://dummy:dummy@",
        "noProxy": ".cluster.local,.svc,,,,,,api-int.ostest.shiftstack.com,localhost"

As stated in the documentation, the proxy must be able to reply to the router that the cluster uses, so below route is added on the proxy server:

sudo nmcli connection modify '{{ iface_con.stdout }}' +ipv4.routes ' {{ restricted_network.default_gw }}

The restricted_network.default_gw should be defined on the subnet and available so the OCP cluster can make use of it.

Comment 6 ShiftStack Bugwatcher 2022-03-05 07:07:13 UTC
Removing the Triaged keyword because:
* the priority assessment is missing
* the QE automation assessment (flag qe_test_coverage) is missing

Comment 8 errata-xmlrpc 2022-03-12 04:36:01 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.