Description of problem: After successfully completion of running ./openshift-install --dir=/root/cloudlet1/ wait-for bootstrap-complete --log-level debug I can't continue with the deployment of the cluster since I have not all cluster operators in True state. Version-Release number of selected component (if applicable): 4.2.4 How reproducible: Done more that once Steps to Reproduce: 1.Following the guide 2.Running ./openshift-install --dir=/root/cloudlet1/ wait-for bootstrap-complete --log-level debug 3.export KUBECONFIG 4. oc whoami 5. oc get nodes [root@ocp41-installer cloudlet1]# oc get nodes NAME STATUS ROLES AGE VERSION control-plane-0.ocp41.sales.lab.tlv.redhat.com Ready master 73m v1.14.6+c7d2111b9 control-plane-1.ocp41.sales.lab.tlv.redhat.com Ready master 73m v1.14.6+c7d2111b9 control-plane-2.ocp41.sales.lab.tlv.redhat.com Ready master 73m v1.14.6+c7d2111b9 localhost Ready,SchedulingDisabled master 78m v1.14.6+c7d2111b9 6. oc get csr NAME AGE REQUESTOR CONDITION csr-249jl 71m system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued csr-2nhr9 71m system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued csr-7x6cv 67m system:node:localhost Approved,Issued csr-94bmj 71m system:node:localhost Approved,Issued csr-95r9b 66m system:node:control-plane-1.ocp41.sales.lab.tlv.redhat.com Approved,Issued csr-9nzjh 71m system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued csr-bk674 70m system:node:localhost Approved,Issued csr-d8vpc 71m system:node:localhost Approved,Issued csr-dtqsk 71m system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued csr-fmqvp 71m system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued csr-grcbb 66m system:node:control-plane-0.ocp41.sales.lab.tlv.redhat.com Approved,Issued csr-jfcvg 71m system:node:localhost Approved,Issued csr-k6dsz 67m system:node:localhost Approved,Issued csr-nq86v 71m system:node:localhost Approved,Issued csr-shz78 71m system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued csr-t4tsq 71m system:node:localhost Approved,Issued csr-w6fz6 66m system:node:control-plane-2.ocp41.sales.lab.tlv.redhat.com Approved,Issued csr-wv5wm 67m system:node:localhost Approved,Issued oc get clusteroperators [root@ocp41-installer cloudlet1]# oc get clusteroperators NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE authentication Unknown Unknown True 59m cloud-credential 4.2.4 True False False 78m cluster-autoscaler 4.2.4 True False False 57m console 4.2.4 Unknown True False 57m dns 4.2.4 True False False 74m image-registry False True False 56m ingress unknown False True True 56m insights 4.2.4 True False False 78m kube-apiserver 4.2.4 True False False 70m kube-controller-manager 4.2.4 True False False 66m kube-scheduler 4.2.4 True False False 70m machine-api 4.2.4 True False False 78m machine-config 4.2.4 False False True 78m marketplace 4.2.4 True False False 58m monitoring False True True 54m network 4.2.4 True False False 72m node-tuning 4.2.4 True False False 58m openshift-apiserver 4.2.4 True False False 58s openshift-controller-manager 4.2.4 True False False 71m openshift-samples 4.2.4 True False False 35m operator-lifecycle-manager 4.2.4 True False False 73m operator-lifecycle-manager-catalog 4.2.4 True False False 73m operator-lifecycle-manager-packageserver 4.2.4 True False False 35m service-ca 4.2.4 True False False 78m service-catalog-apiserver 4.2.4 True False False 59m service-catalog-controller-manager 4.2.4 True False False 59m storage 4.2.4 True False False 56m Actual results: Not all Cluster operators in Available True State and even after applying patch fo image-registry it's state not changing over time. As a result can't complete cluster deployment Fail with error: ./openshift-install --dir=/root/cloudlet1/ wait-for install-complete --log-level debug DEBUG OpenShift Installer v4.2.4 DEBUG Built from commit 425e4ff0037487e32571258640b39f56d5ee5572 INFO Waiting up to 30m0s for the cluster at https://api.ocp41.sales.lab.tlv.redhat.com:6443 to initialize... DEBUG Still waiting for the cluster to initialize: Working towards 4.2.4: 98% complete DEBUG Still waiting for the cluster to initialize: Cluster operator machine-config is reporting a failure: Failed to resync 4.2.4 because: timed out waiting for the condition during syncRequiredMachineConfigPools: error pool master is not ready, retrying. Status: (pool degraded: true total: 4, ready 0, updated: 0, unavailable: 1) Expected results: All cluster operatrors in Available true state, Cluster install-complete should be completed successfully. Additional info: I am trying with 4.2.4 since it was another issue with 4.2.8 version - openshift-install --dir=/root/cloudlet1/ wait-for bootstrap-complete was failed with error : api.domainname.com:6443 was refused
moving to machine-config as it's one of the failing operator.
Hi, I am not so understand which info you need me to provide? Please elaborate little bit more!
@Anoel: please attach the logs using this tool: https://github.com/openshift/must-gather
Hi, I uploaded the tar file with all logs to the Google drive You can find it in the next link https://drive.google.com/file/d/1nCmomYoInn0SWY0dcsmaOV0Te-_UykNz/view?usp=sharing
Looking at the machineconfig pools both master and worker aren't degraded: degradedMachineCount: 0 machineCount: 3 observedGeneration: 2 readyMachineCount: 3 unavailableMachineCount: 0 updatedMachineCount: 3 degradedMachineCount: 0 machineCount: 2 observedGeneration: 2 readyMachineCount: 2 unavailableMachineCount: 0 updatedMachineCount: 2 MCO is also fine: conditions: - lastTransitionTime: 2019-12-18T07:01:48Z message: Cluster has deployed 4.2.4 status: "True" type: Available - lastTransitionTime: 2019-12-18T07:01:48Z message: Cluster version is 4.2.4 status: "False" type: Progressing - lastTransitionTime: 2019-12-18T07:00:35Z status: "False" type: Degraded - lastTransitionTime: 2019-12-18T07:01:48Z reason: AsExpected status: "True" type: Upgradeable
I issue that I do see is: conditions: - lastTransitionTime: 2019-12-18T06:58:59Z status: "False" type: Available - lastTransitionTime: 2019-12-18T07:19:24Z message: Cluster operator image-registry is still updating reason: ClusterOperatorNotAvailable status: "True" type: Failing - lastTransitionTime: 2019-12-18T06:58:59Z message: 'Unable to apply 4.2.4: the cluster operator image-registry has not yet successfully rolled out' reason: ClusterOperatorNotAvailable status: "True" type: Progressing Reassigning: Image-Registry, PTAL
*** This bug has been marked as a duplicate of bug 1770658 ***
Why I am still receiving email every day with this outstanding request? [Red Hat Bugzilla] Your Outstanding Requests Inbox x bugzilla 4:24 AM (6 hours ago) to me The following is a list of bugs or attachments to bugs in which a user has been waiting more than 3 days for a response from you. Please take action on these requests as quickly as possible. (Note that some of these bugs might already be closed, but a user is still waiting for your response.) We'll remind you again tomorrow if these requests are still outstanding, or if there are any new requests where users have been waiting more than 3 days for your response. needinfo -------- Bug 1779196: vSphere UPI: failed to initialize the cluster: Some cluster operators are still updating: authentication, console (19 days old) https://bugzilla.redhat.com/show_bug.cgi?id=1779196