Description of problem: ingress reports degrade with CanaryChecksSucceeding=False in the cluster with proxy setting Version-Release number of selected component (if applicable): 4.7.0-0.nightly-2021-03-04-093123 profile: aos-4_7/ipi-on-aws/versioned-installer-customer_vpc-http_proxy-multiblockdevices-fips-ovn-ipsec-c How reproducible: always Steps to Reproduce: 1. install cluster with the profile: aos-4_7/ipi-on-aws/versioned-installer-customer_vpc-http_proxy-multiblockdevices-fips-ovn-ipsec-ci Actual results: co/ingress reports degrade after installation $ oc get co/ingress NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE ingress 4.7.0-0.nightly-2021-03-04-093123 True False True 91m $ oc get co/ingress -oyaml <---snip---> spec: {} status: conditions: - lastTransitionTime: "2021-03-05T01:48:49Z" message: desired and current number of IngressControllers are equal reason: AsExpected status: "True" type: Available - lastTransitionTime: "2021-03-05T01:48:49Z" message: desired and current number of IngressControllers are equal reason: AsExpected status: "False" type: Progressing - lastTransitionTime: "2021-03-05T01:50:07Z" message: 'Some ingresscontrollers are degraded: ingresscontroller "default" is degraded: DegradedConditions: One or more other status conditions indicate a degraded state: CanaryChecksSucceeding=False (CanaryChecksRepetitiveFailures: Canary route checks for the default ingress controller are failing)' reason: IngressControllersDegraded status: "True" type: Degraded $ oc -n openshift-ingress-operator rsh ingress-operator-6866b54f8b-npcx2 Defaulting container name to ingress-operator. Use 'oc describe pod/ingress-operator-6866b54f8b-npcx2 -n openshift-ingress-operator' to see all of the containers in this pod. sh-4.4$ sh-4.4$ env | grep -i proxy HTTP_PROXY=http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@ec2-18-188-209-51.us-east-2.compute.amazonaws.com:3128 NO_PROXY=.cluster.local,.svc,.us-east-2.compute.internal,10.0.0.0/16,10.128.0.0/14,127.0.0.1,169.254.169.254,172.30.0.0/16,api-int.zhsungre1.qe.devcluster.openshift.com,localhost,test.no-proxy.com HTTPS_PROXY=http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@ec2-18-188-209-51.us-east-2.compute.amazonaws.com:3128 sh-4.4$ sh-4.4$ curl http://canary-openshift-ingress-canary.apps.zhsungre1.qe.devcluster.openshift.com/ -v * Uses proxy env variable NO_PROXY == '.cluster.local,.svc,.us-east-2.compute.internal,10.0.0.0/16,10.128.0.0/14,127.0.0.1,169.254.169.254,172.30.0.0/16,api-int.zhsungre1.qe.devcluster.openshift.com,localhost,test.no-proxy.com' * Trying 3.18.114.75... * TCP_NODELAY set ------->>>>>>> ( time out eventually ) sh-4.4$ curl https://canary-openshift-ingress-canary.apps.zhsungre1.qe.devcluster.openshift.com/ -v * Uses proxy env variable NO_PROXY == '.cluster.local,.svc,.us-east-2.compute.internal,10.0.0.0/16,10.128.0.0/14,127.0.0.1,169.254.169.254,172.30.0.0/16,api-int.zhsungre1.qe.devcluster.openshift.com,localhost,test.no-proxy.com' * Uses proxy env variable HTTPS_PROXY == 'http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@ec2-18-188-209-51.us-east-2.compute.amazonaws.com:3128' * Trying 10.0.3.228... * TCP_NODELAY set * Connected to ec2-18-188-209-51.us-east-2.compute.amazonaws.com (10.0.3.228) port 3128 (#0) Expected results: co/ingress should not report degraded in such cluster Additional info: seems "curl http://canary..." didn't use the proxy and time out, but "curl https://canary..." used the proxy and can work
Tested same version on Azure/GCP with proxy setting, but cannot reproduce the issue. $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.7.0-0.nightly-2021-03-04-093123 True False 28m Cluster version is 4.7.0-0.nightly-2021-03-04-093123 $ oc get infrastructures.config.openshift.io cluster -oyaml spec: cloudConfig: key: config name: cloud-provider-config platformSpec: type: GCP status: apiServerInternalURI: https://api-int.hongli-gcpxy.qe.gcp.devcluster.openshift.com:6443 apiServerURL: https://api.hongli-gcpxy.qe.gcp.devcluster.openshift.com:6443 etcdDiscoveryDomain: "" infrastructureName: hongli-gcpxy-dchd6 platform: GCP platformStatus: gcp: projectID: openshift-qe region: us-central1 type: GCP $ oc get proxies.config.openshift.io cluster -oyaml spec: httpProxy: http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@10.0.0.2:3128 httpsProxy: http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@10.0.0.2:3128 noProxy: test.no-proxy.com trustedCA: name: "" status: httpProxy: http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@10.0.0.2:3128 httpsProxy: http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@10.0.0.2:3128 noProxy: .cluster.local,.svc,10.0.0.0/16,10.128.0.0/14,127.0.0.1,169.254.169.254,172.30.0.0/16,api-int.hongli-gcpxy.qe.gcp.devcluster.openshift.com,localhost,metadata,metadata.google.internal,metadata.google.internal.,test.no-proxy.com $ oc -n openshift-ingress-canary get route NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD canary canary-openshift-ingress-canary.apps.hongli-gcpxy.qe.gcp.devcluster.openshift.com ingress-canary 8080 edge/Redirect None $ oc -n openshift-ingress-operator rsh ingress-operator-58c955ffd4-tkxnr Defaulting container name to ingress-operator. Use 'oc describe pod/ingress-operator-58c955ffd4-tkxnr -n openshift-ingress-operator' to see all of the containers in this pod. sh-4.4$ sh-4.4$ sh-4.4$ env | grep -i proxy HTTP_PROXY=http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@10.0.0.2:3128 NO_PROXY=.cluster.local,.svc,10.0.0.0/16,10.128.0.0/14,127.0.0.1,169.254.169.254,172.30.0.0/16,api-int.hongli-gcpxy.qe.gcp.devcluster.openshift.com,localhost,metadata,metadata.google.internal,metadata.google.internal.,test.no-proxy.com HTTPS_PROXY=http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@10.0.0.2:3128 sh-4.4$ sh-4.4$ curl http://canary-openshift-ingress-canary.apps.hongli-gcpxy.qe.gcp.devcluster.openshift.com -v -kL * Rebuilt URL to: http://canary-openshift-ingress-canary.apps.hongli-gcpxy.qe.gcp.devcluster.openshift.com/ * Uses proxy env variable NO_PROXY == '.cluster.local,.svc,10.0.0.0/16,10.128.0.0/14,127.0.0.1,169.254.169.254,172.30.0.0/16,api-int.hongli-gcpxy.qe.gcp.devcluster.openshift.com,localhost,metadata,metadata.google.internal,metadata.google.internal.,test.no-proxy.com' * Trying 130.211.196.23... * TCP_NODELAY set * Connected to canary-openshift-ingress-canary.apps.hongli-gcpxy.qe.gcp.devcluster.openshift.com (130.211.196.23) port 80 (#0) > GET / HTTP/1.1 > Host: canary-openshift-ingress-canary.apps.hongli-gcpxy.qe.gcp.devcluster.openshift.com > User-Agent: curl/7.61.1 > Accept: */* > < HTTP/1.1 302 Found < Cache-Control: no-cache < Content-length: 0 < Location: https://canary-openshift-ingress-canary.apps.hongli-gcpxy.qe.gcp.devcluster.openshift.com/ < * Connection #0 to host canary-openshift-ingress-canary.apps.hongli-gcpxy.qe.gcp.devcluster.openshift.com left intact * Issue another request to this URL: 'https://canary-openshift-ingress-canary.apps.hongli-gcpxy.qe.gcp.devcluster.openshift.com/' * Uses proxy env variable NO_PROXY == '.cluster.local,.svc,10.0.0.0/16,10.128.0.0/14,127.0.0.1,169.254.169.254,172.30.0.0/16,api-int.hongli-gcpxy.qe.gcp.devcluster.openshift.com,localhost,metadata,metadata.google.internal,metadata.google.internal.,test.no-proxy.com' * Uses proxy env variable HTTPS_PROXY == 'http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@10.0.0.2:3128' * Trying 10.0.0.2... * TCP_NODELAY set * Connected to 10.0.0.2 (10.0.0.2) port 3128 (#1) * allocate connect buffer! * Establish HTTP proxy tunnel to canary-openshift-ingress-canary.apps.hongli-gcpxy.qe.gcp.devcluster.openshift.com:443 * Proxy auth using Basic with user 'proxy-user1' > CONNECT canary-openshift-ingress-canary.apps.hongli-gcpxy.qe.gcp.devcluster.openshift.com:443 HTTP/1.1 > Host: canary-openshift-ingress-canary.apps.hongli-gcpxy.qe.gcp.devcluster.openshift.com:443 > Proxy-Authorization: Basic cHJveHktdXNlcjE6SllnVThxUlpWNERZNFBYSmJ4Sks= > User-Agent: curl/7.61.1 > Proxy-Connection: Keep-Alive > < HTTP/1.1 200 Connection established < * Proxy replied 200 to CONNECT request * CONNECT phase completed! * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /etc/pki/tls/certs/ca-bundle.crt CApath: none * TLSv1.3 (OUT), TLS handshake, Client hello (1): * CONNECT phase completed! * CONNECT phase completed! * TLSv1.3 (IN), TLS handshake, Server hello (2): * TLSv1.3 (IN), TLS handshake, [no content] (0): * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): * TLSv1.3 (IN), TLS handshake, [no content] (0): * TLSv1.3 (IN), TLS handshake, Certificate (11): * TLSv1.3 (IN), TLS handshake, [no content] (0): * TLSv1.3 (IN), TLS handshake, CERT verify (15): * TLSv1.3 (IN), TLS handshake, [no content] (0): * TLSv1.3 (IN), TLS handshake, Finished (20): * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): * TLSv1.3 (OUT), TLS handshake, [no content] (0): * TLSv1.3 (OUT), TLS handshake, Finished (20): * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 * ALPN, server did not agree to a protocol * Server certificate: * subject: CN=*.apps.hongli-gcpxy.qe.gcp.devcluster.openshift.com * start date: Mar 5 07:42:38 2021 GMT * expire date: Mar 5 07:42:39 2023 GMT * issuer: CN=ingress-operator@1614930157 * SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway. * TLSv1.3 (OUT), TLS app data, [no content] (0): > GET / HTTP/1.1 > Host: canary-openshift-ingress-canary.apps.hongli-gcpxy.qe.gcp.devcluster.openshift.com > User-Agent: curl/7.61.1 > Accept: */* > * TLSv1.3 (IN), TLS handshake, [no content] (0): * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): * TLSv1.3 (IN), TLS handshake, [no content] (0): * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): * TLSv1.3 (IN), TLS app data, [no content] (0): < HTTP/1.1 200 OK < X-Request-Port: 8080 < Date: Fri, 05 Mar 2021 08:39:04 GMT < Content-Length: 17 < Content-Type: text/plain; charset=utf-8 < Set-Cookie: c6e529a6ab19a530fd4f1cceb91c08a9=79d2a53e9db507e1f33d020ea2b01580; path=/; HttpOnly; Secure; SameSite=None < Cache-control: private < Hello OpenShift! * Connection #1 to host 10.0.0.2 left intact sh-4.4$ sh-4.4$ sh-4.4$ curl https://canary-openshift-ingress-canary.apps.hongli-gcpxy.qe.gcp.devcluster.openshift.com -v -kL * Rebuilt URL to: https://canary-openshift-ingress-canary.apps.hongli-gcpxy.qe.gcp.devcluster.openshift.com/ * Uses proxy env variable NO_PROXY == '.cluster.local,.svc,10.0.0.0/16,10.128.0.0/14,127.0.0.1,169.254.169.254,172.30.0.0/16,api-int.hongli-gcpxy.qe.gcp.devcluster.openshift.com,localhost,metadata,metadata.google.internal,metadata.google.internal.,test.no-proxy.com' * Uses proxy env variable HTTPS_PROXY == 'http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@10.0.0.2:3128' * Trying 10.0.0.2... * TCP_NODELAY set * Connected to 10.0.0.2 (10.0.0.2) port 3128 (#0) * allocate connect buffer! * Establish HTTP proxy tunnel to canary-openshift-ingress-canary.apps.hongli-gcpxy.qe.gcp.devcluster.openshift.com:443 * Proxy auth using Basic with user 'proxy-user1' > CONNECT canary-openshift-ingress-canary.apps.hongli-gcpxy.qe.gcp.devcluster.openshift.com:443 HTTP/1.1 > Host: canary-openshift-ingress-canary.apps.hongli-gcpxy.qe.gcp.devcluster.openshift.com:443 > Proxy-Authorization: Basic cHJveHktdXNlcjE6SllnVThxUlpWNERZNFBYSmJ4Sks= > User-Agent: curl/7.61.1 > Proxy-Connection: Keep-Alive > < HTTP/1.1 200 Connection established < * Proxy replied 200 to CONNECT request * CONNECT phase completed! * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /etc/pki/tls/certs/ca-bundle.crt CApath: none * TLSv1.3 (OUT), TLS handshake, Client hello (1): * CONNECT phase completed! * CONNECT phase completed! * TLSv1.3 (IN), TLS handshake, Server hello (2): * TLSv1.3 (IN), TLS handshake, [no content] (0): * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): * TLSv1.3 (IN), TLS handshake, [no content] (0): * TLSv1.3 (IN), TLS handshake, Certificate (11): * TLSv1.3 (IN), TLS handshake, [no content] (0): * TLSv1.3 (IN), TLS handshake, CERT verify (15): * TLSv1.3 (IN), TLS handshake, [no content] (0): * TLSv1.3 (IN), TLS handshake, Finished (20): * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): * TLSv1.3 (OUT), TLS handshake, [no content] (0): * TLSv1.3 (OUT), TLS handshake, Finished (20): * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 * ALPN, server did not agree to a protocol * Server certificate: * subject: CN=*.apps.hongli-gcpxy.qe.gcp.devcluster.openshift.com * start date: Mar 5 07:42:38 2021 GMT * expire date: Mar 5 07:42:39 2023 GMT * issuer: CN=ingress-operator@1614930157 * SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway. * TLSv1.3 (OUT), TLS app data, [no content] (0): > GET / HTTP/1.1 > Host: canary-openshift-ingress-canary.apps.hongli-gcpxy.qe.gcp.devcluster.openshift.com > User-Agent: curl/7.61.1 > Accept: */* > * TLSv1.3 (IN), TLS handshake, [no content] (0): * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): * TLSv1.3 (IN), TLS handshake, [no content] (0): * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): * TLSv1.3 (IN), TLS app data, [no content] (0): < HTTP/1.1 200 OK < X-Request-Port: 8080 < Date: Fri, 05 Mar 2021 08:39:14 GMT < Content-Length: 17 < Content-Type: text/plain; charset=utf-8 < Set-Cookie: c6e529a6ab19a530fd4f1cceb91c08a9=46e9977a9255b48ae1413425f8ae159f; path=/; HttpOnly; Secure; SameSite=None < Cache-control: private < Hello OpenShift! * Connection #0 to host 10.0.0.2 left intact sh-4.4$
Thank you for your detailed explanation. @Stephen verified with 4.8.0-0.nightly-2021-03-08-053437 and passed. $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.8.0-0.nightly-2021-03-08-053437 True False 47m Cluster version is 4.8.0-0.nightly-2021-03-08-053437 $ oc get co/ingress NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE ingress 4.8.0-0.nightly-2021-03-08-053437 True False False 61m $ oc get proxy cluster -oyaml <---spec: httpProxy: http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@ec2-3-142-53-28.us-east-2.compute.amazonaws.com:3128 httpsProxy: http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@ec2-3-142-53-28.us-east-2.compute.amazonaws.com:3128 noProxy: test.no-proxy.com trustedCA: name: "" status: httpProxy: http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@ec2-3-142-53-28.us-east-2.compute.amazonaws.com:3128 httpsProxy: http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@ec2-3-142-53-28.us-east-2.compute.amazonaws.com:3128 noProxy: .cluster.local,.svc,.us-east-2.compute.internal,10.0.0.0/16,10.128.0.0/14,127.0.0.1,169.254.169.254,172.30.0.0/16,api-int.hongli-np.qe.devcluster.openshift.com,localhost,test.no-proxy.comsnip--->
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:2438