Bug 1972417
| Summary: | [Assisted-4.8] [Staging] OCS and CNV operators deployments fails for cluster that require HTTP_PROXY | ||||||
|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Lital Alon <lalon> | ||||
| Component: | assisted-installer | Assignee: | Piotr Kliczewski <pkliczew> | ||||
| assisted-installer sub component: | CNV Plugin | QA Contact: | Lital Alon <lalon> | ||||
| Status: | CLOSED NOTABUG | Docs Contact: | |||||
| Severity: | urgent | ||||||
| Priority: | unspecified | CC: | aos-bugs, masayag, nshidlin | ||||
| Version: | 4.8 | Flags: | lalon:
needinfo-
lalon: needinfo- |
||||
| Target Milestone: | --- | ||||||
| Target Release: | --- | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2021-06-16 11:44:07 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
|
Description
Lital Alon
2021-06-15 20:23:07 UTC
Moti, I remember you work on the proxy in the past. Do you know why some nodes are unable to pull the images? The link to https://qaprodauth.cloud.redhat.com/openshift/assisted-installer/clusters/77bdeb31-7112-4636-a925-a5f8ae193527 is unavaiable. However, gathered more information from the installed cluster: [root@sealusa48 ~]# oc get proxy cluster -o yaml apiVersion: config.openshift.io/v1 kind: Proxy metadata: creationTimestamp: "2021-06-15T17:31:25Z" generation: 1 manager: cluster-bootstrap operation: Update time: "2021-06-15T17:31:26Z" name: cluster resourceVersion: "471" uid: 6277ca16-4a24-4238-a264-f8e400a8b1b5 spec: httpProxy: http://assisted:redhat@10.19.131.5:59151 httpsProxy: http://assisted:redhat@10.19.131.5:59151 noProxy: 192.168.127.0/24,.test-infra-cluster-c912ff98.qe.lab.redhat.com,10.128.0.0/14,172.30.0.0/16 trustedCA: name: user-ca-bundle status: httpProxy: http://assisted:redhat@10.19.131.5:59151 httpsProxy: http://assisted:redhat@10.19.131.5:59151 noProxy: .cluster.local,.svc,.test-infra-cluster-c912ff98.qe.lab.redhat.com,10.128.0.0/14,127.0.0.1,172.30.0.0/16,192.168.127.0/24,api-int.test-infra-cluster-c912ff98.qe.lab.redhat.com,localhost and this issue is not limited only to OCS and CNV namespaces: [root@sealusa48 ~]# oc get pods -A | grep ImagePullBackOff openshift-cnv cdi-deployment-67cd58976-xzpnp 0/1 ImagePullBackOff 0 12h openshift-cnv kubevirt-node-labeller-rxg8l 0/1 Init:ImagePullBackOff 0 12h openshift-cnv kubevirt-node-labeller-s9h85 0/1 Init:ImagePullBackOff 0 12h openshift-cnv kubevirt-node-labeller-vw9pj 0/1 Init:ImagePullBackOff 0 12h openshift-cnv virt-api-6cbf698cd-2xhl4 0/1 ImagePullBackOff 0 12h openshift-cnv virt-api-6cbf698cd-t9zcv 0/1 ImagePullBackOff 0 12h openshift-cnv vm-import-controller-54bd9d975b-zrwns 0/1 ImagePullBackOff 0 12h openshift-marketplace certified-operators-n8mxp 0/1 ImagePullBackOff 0 12h openshift-marketplace community-operators-vsg9q 0/1 ImagePullBackOff 0 12h openshift-marketplace redhat-marketplace-fss7m 0/1 ImagePullBackOff 0 12h openshift-marketplace redhat-operators-f76z9 0/1 ImagePullBackOff 0 12h openshift-monitoring prometheus-adapter-648d79bf67-k7b6r 0/1 ImagePullBackOff 0 68m openshift-storage csi-cephfsplugin-8tfvr 1/3 ImagePullBackOff 0 12h openshift-storage csi-cephfsplugin-provisioner-6685dc595c-bvvtp 2/6 ImagePullBackOff 0 12h openshift-storage csi-cephfsplugin-provisioner-6685dc595c-wfhqc 3/6 ImagePullBackOff 0 12h openshift-storage csi-cephfsplugin-t2j76 1/3 ImagePullBackOff 0 12h openshift-storage csi-rbdplugin-provisioner-7d98bc744-455nx 2/6 ImagePullBackOff 0 12h openshift-storage csi-rbdplugin-provisioner-7d98bc744-vbnpf 1/6 ImagePullBackOff 0 12h openshift-storage csi-rbdplugin-v55gl 1/3 ImagePullBackOff 0 12h openshift-storage csi-rbdplugin-wctd5 1/3 ImagePullBackOff 0 12h openshift-storage rook-ceph-detect-version-ckgg4 0/1 ImagePullBackOff 0 20m Viewing the events of pod openshift-cnv/cdi-deployment-67cd58976-xzpnp: Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Pulling 168m (x121 over 12h) kubelet Pulling image "registry.redhat.io/container-native-virtualization/virt-cdi-controller@sha256:3165e1e22341c54170dc0d3413b121d7498bca7c10ee3dc38b819ed6ec0b590f" Normal BackOff 3m29s (x3350 over 12h) kubelet Back-off pulling image "registry.redhat.io/container-native-virtualization/virt-cdi-controller@sha256:3165e1e22341c54170dc0d3413b121d7498bca7c10ee3dc38b819ed6ec0b590f" [root@sealusa48 ~]# ping registry.redhat.io PING e14353.g.akamaiedge.net (23.51.138.73) 56(84) bytes of data. After investigation we got to conclusion that this is not a real bug but test issue. This bug reproduced by test_infra - but test failed, due to operator deployment failure caused by a bug that doesn't related to HTTP_PROXY (https://bugzilla.redhat.com/show_bug.cgi?id=1968181). This failure caused the test-infra container to finish, which caused the HTTP proxy server to be down (because the squad container, AKA proxy server, is starting up from within test-infra container). Both OCS and CNV were in fact Progressing, and proxy was down, which caused pull image error. Added a test to cover OCS CNV and HTTP proxy to test infra, but it wont run properly until mentioned OCS bug will be fixed (https://gitlab.cee.redhat.com/ocp-edge-qe/kni-assisted-installer-auto/-/merge_requests/354) Closing the bug. |