Bug 1862948
| Summary: | [QE build watch] Upgrade stuck in 'failed to run command oc (6 tries): timed out waiting for the condition: running oc image extract --path /:/run/mco-machine-os-content/os-content-176084702 ... Timeout exceeded while awaiting headers' | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Xingxing Xia <xxia> |
| Component: | Cluster Version Operator | Assignee: | Lalatendu Mohanty <lmohanty> |
| Status: | CLOSED DUPLICATE | QA Contact: | Johnny Liu <jialiu> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 4.6 | CC: | aos-bugs, jokerman, kewang, knarra, skumari, vrutkovs, wking, xtian |
| Target Milestone: | --- | Keywords: | Reopened |
| Target Release: | 4.6.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-08-11 07:49:51 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Xingxing Xia
2020-08-03 09:43:20 UTC
*** This bug has been marked as a duplicate of bug 1862979 *** (In reply to Xingxing Xia from comment #0) > Description of problem: > Matrix: 4.5.4-x86_64 -> 4.6.0-0.nightly-2020-08-01-172303, 15_Disconnected UPI on GCP with RHEL7.7 OVN & http_proxy & Etcd Encryption on This bug env had http_proxy, thus not a pure disconnected env bug as bug 1862979 which is pure disconnected env without proxy. And the error message is a different. This bug error is "Timeout exceeded while awaiting headers", while that bug is "Get "https://quay.io/v2/": Forbidden". So I don't think it is same issue. Reopenning therefore... Thanks. (In reply to Xingxing Xia from comment #3) > This bug env had http_proxy One more clearer point, with http_proxy, this bug env does not use "mirror registry" like bug 1862979 at all to host the images. (In reply to Xingxing Xia from comment #3) > (In reply to Xingxing Xia from comment #0) > > Description of problem: > > Matrix: 4.5.4-x86_64 -> 4.6.0-0.nightly-2020-08-01-172303, 15_Disconnected UPI on GCP with RHEL7.7 OVN & http_proxy & Etcd Encryption on > This bug env had http_proxy, thus not a pure disconnected env bug as bug > 1862979 which is pure disconnected env without proxy. And the error message > is a different. This bug error is "Timeout exceeded while awaiting headers", > while that bug is "Get "https://quay.io/v2/": Forbidden". So I don't think > it is same issue. Reopenning therefore... > Thanks. Please attach the must-gather. So far the bug cause is the same - MCO is now using `oc image extract` instead of `podman run`, which means mirror and proxy settings are not respected Unlike other failed QE CI jobs with successful must-gather, this QE CI job shown above had failed must-gather result "must-gather file creation fails". Will rebuild it and keep the cluster and come back. Reproduced in QE build upgrade testing: http://10.0.76.54/buildcorp/upgrade_CI/4207/console Cluster profile: Disconnected IPI on Azure & Private Cluster From 4.5.4 to 4.6 nightly build, oc adm upgrade --to-image=ugdci05072549.mirror-registry.qe.azure.devcluster.openshift.com:5000/openshift-release-dev/ocp-release:4.6.0-0.nightly-2020-08-04-210224 --force=true --allow-explicit-upgrade=true Hit the same problem, ... Status: Conditions: Last Transition Time: 2020-08-05T02:43:01Z Message: Working towards 4.6.0-0.nightly-2020-08-04-210224 Status: True Type: Progressing Last Transition Time: 2020-08-05T02:57:45Z Message: Unable to apply 4.6.0-0.nightly-2020-08-04-210224: timed out waiting for the condition during syncRequiredMachineConfigPools: pool master has not progressed to latest configuration: controller version mismatch for rendered-master-09c695ebfc8d928626a4c4b84684d89a expected ade383fc8b27be6bdc6aa7985b3154350beaec88 has 99eb744f5094224edb60d88ca85d607ab151ebdf: pool is degraded because nodes fail with "1 nodes are reporting degraded status on sync": "Node ugdci05072549-x2q48-master-2 is reporting: \"failed to run command oc (6 tries): timed out waiting for the condition: running oc image extract --path /:/run/mco-machine-os-content/os-content-424723676 --registry-config /var/lib/kubelet/config.json quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16a8dde4b893ff0b5b4aeb05474f2f5e2ce9cac45d5d3e98b40c4309e23215a7 failed: error: unable to connect to image repository quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16a8dde4b893ff0b5b4aeb05474f2f5e2ce9cac45d5d3e98b40c4309e23215a7: Get https://quay.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) n: exit status 1\"", retrying Reason: RequiredPoolsFailed Status: True Type: Degraded Last Transition Time: 2020-08-05T02:41:04Z Message: Cluster not available for 4.6.0-0.nightly-2020-08-04-210224 ... Reproduced the issue in QE CI upgrade testing. Below are the details:
Build details : 4.5.5-x86_64 -> 4.6.0-0.nightly-2020-08-05-153221
Matrix : 14_Disconnected IPI on Azure & Private Cluster
Upgrade command: ./oc adm upgrade --to-image=ugdci06022546.mirror-registry.qe.azure.devcluster.openshift.com:5000/openshift-release-dev/ocp-release:4.6.0-0.nightly-2020-08-05-153221 --force=true --allow-explicit-upgrade=true
Hit the same problem:
========================
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Abnormal node details~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Name: ugdci06022546-lk2z9-master-2
Roles: master
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/instance-type=Standard_D8s_v3
beta.kubernetes.io/os=linux
failure-domain.beta.kubernetes.io/region=centralus
failure-domain.beta.kubernetes.io/zone=centralus-1
kubernetes.io/arch=amd64
kubernetes.io/hostname=ugdci06022546-lk2z9-master-2
kubernetes.io/os=linux
node-role.kubernetes.io/master=
node.kubernetes.io/instance-type=Standard_D8s_v3
node.openshift.io/os_id=rhcos
topology.kubernetes.io/region=centralus
topology.kubernetes.io/zone=centralus-1
Annotations: machine.openshift.io/machine: openshift-machine-api/ugdci06022546-lk2z9-master-2
machineconfiguration.openshift.io/currentConfig: rendered-master-7ff4949bb27c168d12c4ed000abfbea5
machineconfiguration.openshift.io/desiredConfig: rendered-master-e805edb843a99dd5f5f82c9c07789565
machineconfiguration.openshift.io/reason:
failed to run command oc (6 tries): timed out waiting for the condition: running oc image extract --path /:/run/mco-machine-os-content/os-...
: exit status 1
machineconfiguration.openshift.io/state: Degraded
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Wed, 05 Aug 2020 15:02:27 -0400
Taints: node-role.kubernetes.io/master:NoSchedule
node.kubernetes.io/unschedulable:NoSchedule
Unschedulable: true
Lease:
HolderIdentity: ugdci06022546-lk2z9-master-2
Last Sync Error: pool master has not progressed to latest configuration: controller version mismatch for rendered-master-7ff4949bb27c168d12c4ed000abfbea5 expected ade383fc8b27be6bdc6aa7985b3154350beaec88 has 807abb900cf9976a1baad66eab17c6d76016e7b7: pool is degraded because nodes fail with "1 nodes are reporting degraded status on sync": "Node ugdci06022546-lk2z9-master-2 is reporting: \"failed to run command oc (6 tries): timed out waiting for the condition: running oc image extract --path /:/run/mco-machine-os-content/os-content-308854769 --registry-config /var/lib/kubelet/config.json quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16a8dde4b893ff0b5b4aeb05474f2f5e2ce9cac45d5d3e98b40c4309e23215a7 failed: error: unable to connect to image repository quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16a8dde4b893ff0b5b4aeb05474f2f5e2ce9cac45d5d3e98b40c4309e23215a7: Get https://quay.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\\n: exit status 1\"", retrying
Hit similar issue on another matrix and upgrade path, below are the details:
Build Details: 4.5.5-x86_64 -> 4.6.0-0.nightly-2020-08-05-174122
Matrix : 26_Disconnected IPI on OSP13 with https_proxy & Etcd Encryption on
oc adm upgrade --to-image=registry.svc.ci.openshift.org/ocp/release:4.6.0-0.nightly-2020-08-05-174122 --force=true --allow-explicit-upgrade=true
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Abnormal node details~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Name: ugdci06040054-dl2kr-master-0
Roles: master
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/instance-type=m1.xlarge
beta.kubernetes.io/os=linux
failure-domain.beta.kubernetes.io/region=regionOne
failure-domain.beta.kubernetes.io/zone=nova
kubernetes.io/arch=amd64
kubernetes.io/hostname=ugdci06040054-dl2kr-master-0
kubernetes.io/os=linux
node-role.kubernetes.io/master=
node.kubernetes.io/instance-type=m1.xlarge
node.openshift.io/os_id=rhcos
topology.kubernetes.io/region=regionOne
topology.kubernetes.io/zone=nova
Annotations: machine.openshift.io/machine: openshift-machine-api/ugdci06040054-dl2kr-master-0
machineconfiguration.openshift.io/currentConfig: rendered-master-af74a630fea233a531a58c2184fcaa29
machineconfiguration.openshift.io/desiredConfig: rendered-master-c1501d842176d612c7c001810cd00069
machineconfiguration.openshift.io/reason:
failed to run command oc (6 tries): timed out waiting for the condition: running oc image extract --path /:/run/mco-machine-os-content/os-...
: exit status 1
machineconfiguration.openshift.io/state: Degraded
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Wed, 05 Aug 2020 16:15:35 -0400
Taints: node-role.kubernetes.io/master:NoSchedule
node.kubernetes.io/unschedulable:NoSchedule
Unschedulable: true
Lease:
HolderIdentity: ugdci06040054-dl2kr-master-0
AcquireTime: <unset>
RenewTime: Wed, 05 Aug 2020 20:02:29 -0400
Conditions:
Message: Unable to apply 4.6.0-0.nightly-2020-08-05-174122: timed out waiting for the condition during syncRequiredMachineConfigPools: pool master has not progressed to latest configuration: controller version mismatch for rendered-master-af74a630fea233a531a58c2184fcaa29 expected ade383fc8b27be6bdc6aa7985b3154350beaec88 has 807abb900cf9976a1baad66eab17c6d76016e7b7: pool is degraded because nodes fail with "1 nodes are reporting degraded status on sync": "Node ugdci06040054-dl2kr-master-0 is reporting: \"failed to run command oc (6 tries): timed out waiting for the condition: running oc image extract --path /:/run/mco-machine-os-content/os-content-507018740 --registry-config /var/lib/kubelet/config.json quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16a8dde4b893ff0b5b4aeb05474f2f5e2ce9cac45d5d3e98b40c4309e23215a7 failed: error: unable to connect to image repository quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16a8dde4b893ff0b5b4aeb05474f2f5e2ce9cac45d5d3e98b40c4309e23215a7: Get https://quay.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\\n: exit status 1\"", retrying
Reason: RequiredPoolsFailed
Status: True
must-gather provided in comment #9 shows the same behaviour as in bug 1862979: 2020-08-06T03:35:16.45502974Z error: unable to connect to image repository quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bb8cb875ed5ef903df8f3f056a3d48eaf4cca3b34af02a9d6728125ff507bcdc: Get https://quay.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) 2020-08-06T03:35:16.457232233Z W0806 03:35:16.457158 264417 run.go:44] oc failed: running oc image extract --path /:/run/mco-machine-os-content/os-content-688451146 --registry-config /var/lib/kubelet/config.json quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bb8cb875ed5ef903df8f3f056a3d48eaf4cca3b34af02a9d6728125ff507bcdc failed: error: unable to connect to image repository quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bb8cb875ed5ef903df8f3f056a3d48eaf4cca3b34af02a9d6728125ff507bcdc: Get https://quay.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) Updated the bug to mention proxy environments too *** This bug has been marked as a duplicate of bug 1862979 *** See https://bugzilla.redhat.com/show_bug.cgi?id=1862979#c19 *** This bug has been marked as a duplicate of bug 1862979 *** For future reference, proxy issue was fixed in bug https://bugzilla.redhat.com/show_bug.cgi?id=1857162 |