Bug 1823720 - upi install on azure Cluster operator openshift-apiserver Available is False
Summary: upi install on azure Cluster operator openshift-apiserver Available is False
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: kube-apiserver
Version: 4.3.z
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: 4.3.z
Assignee: Stefan Schimanski
QA Contact: Xingxing Xia
URL:
Whiteboard:
Depends On:
Blocks: 1817717
TreeView+ depends on / blocked
 
Reported: 2020-04-14 10:43 UTC by Ke Wang
Modified: 2020-04-17 08:23 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-04-17 08:23:37 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Comment 1 Stefan Schimanski 2020-04-14 14:27:04 UTC
Please run must-gather and attach the output, if that's possible in that cluster state.

CVO logs would be most interesting here.

Comment 2 Etienne Simard 2020-04-14 15:33:36 UTC
Hello Stefan,

This is a disconnected environment with an offline registry. Without Internet access on that server the must-gather would fail (opened https://bugzilla.redhat.com/show_bug.cgi?id=1823839 for this):

~~~
$ oc adm must-gather
[must-gather      ] OUT unable to resolve the imagestream tag openshift/must-gather:latest
[must-gather      ] OUT 
[must-gather      ] OUT Using must-gather plugin-in image: quay.io/openshift/origin-must-gather:latest
[must-gather      ] OUT namespace/openshift-must-gather-t7zlg created
[must-gather      ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-gwpdf created
[must-gather      ] OUT pod for plug-in image quay.io/openshift/origin-must-gather:latest created
[must-gather-ldn97] OUT gather did not start: unable to pull image: ErrImagePull: rpc error: code = Unknown desc = error pinging docker registry quay.io: Get https://quay.io/v2/: dial tcp 18.208.7.119:443: i/o timeout
[must-gather      ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-gwpdf deleted
[must-gather      ] OUT namespace/openshift-must-gather-t7zlg deleted
error: gather did not start for pod must-gather-ldn97: unable to pull image: ErrImagePull: rpc error: code = Unknown desc = error pinging docker registry quay.io: Get https://quay.io/v2/: dial tcp 18.208.7.119:443: i/o timeout
~~~

I was able to enable the Internet on this cluster to gather the information, but this might not fully preserve the cluster state.

The cluster operators still show the same status:

$ ./oc get co
NAME                                       VERSION                             AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                             4.3.0-0.nightly-2020-04-13-190424   True        False         False      8h
cloud-credential                           4.3.0-0.nightly-2020-04-13-190424   True        False         False      9h
cluster-autoscaler                         4.3.0-0.nightly-2020-04-13-190424   True        False         False      9h
console                                    4.3.0-0.nightly-2020-04-13-190424   True        False         False      8h
dns                                        4.3.0-0.nightly-2020-04-13-190424   True        False         False      9h
image-registry                             4.3.0-0.nightly-2020-04-13-190424   True        False         False      8h
ingress                                    4.3.0-0.nightly-2020-04-13-190424   True        False         False      8h
insights                                   4.3.0-0.nightly-2020-04-13-190424   True        False         False      9h
kube-apiserver                             4.3.0-0.nightly-2020-04-13-190424   True        False         False      9h
kube-controller-manager                    4.3.0-0.nightly-2020-04-13-190424   True        False         False      9h
kube-scheduler                             4.3.0-0.nightly-2020-04-13-190424   True        False         False      9h
machine-api                                4.3.0-0.nightly-2020-04-13-190424   True        False         False      9h
machine-config                             4.3.0-0.nightly-2020-04-13-190424   True        False         False      9h
marketplace                                4.3.0-0.nightly-2020-04-13-190424   True        False         False      9h
monitoring                                 4.3.0-0.nightly-2020-04-13-190424   False       True          True       34m
network                                    4.3.0-0.nightly-2020-04-13-190424   True        False         False      9h
node-tuning                                4.3.0-0.nightly-2020-04-13-190424   True        False         False      9h
openshift-apiserver                        4.3.0-0.nightly-2020-04-13-190424   False       False         False      65m
openshift-controller-manager               4.3.0-0.nightly-2020-04-13-190424   True        False         False      9h
openshift-samples                          4.3.0-0.nightly-2020-04-13-190424   True        False         False      8h
operator-lifecycle-manager                 4.3.0-0.nightly-2020-04-13-190424   True        False         False      9h
operator-lifecycle-manager-catalog         4.3.0-0.nightly-2020-04-13-190424   True        False         False      9h
operator-lifecycle-manager-packageserver   4.3.0-0.nightly-2020-04-13-190424   True        False         False      3h4m
service-ca                                 4.3.0-0.nightly-2020-04-13-190424   True        False         False      9h
service-catalog-apiserver                  4.3.0-0.nightly-2020-04-13-190424   True        False         False      9h
service-catalog-controller-manager         4.3.0-0.nightly-2020-04-13-190424   True        False         False      9h
storage                                    4.3.0-0.nightly-2020-04-13-190424   True        False         False      9h

Comment 4 Ke Wang 2020-04-17 08:23:37 UTC
Unable to reproduce this bug for UPI install on azure, close it.

$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.3.13    True        False         67m     Cluster version is 4.3.13

$ oc get nodes
NAME                                      STATUS   ROLES    AGE   VERSION
keazure4313-04170145-master-0             Ready    master   98m   v1.16.2
keazure4313-04170145-master-1             Ready    master   98m   v1.16.2
keazure4313-04170145-master-2             Ready    master   98m   v1.16.2
keazure4313-04170145-worker-centralus-1   Ready    worker   79m   v1.16.2
keazure4313-04170145-worker-centralus-2   Ready    worker   79m   v1.16.2
keazure4313-04170145-worker-centralus-3   Ready    worker   79m   v1.16.2

$ oc get co
NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                             4.3.13    True        False         False      70m
cloud-credential                           4.3.13    True        False         False      99m
cluster-autoscaler                         4.3.13    True        False         False      88m
console                                    4.3.13    True        False         False      34m
dns                                        4.3.13    True        False         False      94m
image-registry                             4.3.13    True        False         False      50m
ingress                                    4.3.13    True        False         False      37m
insights                                   4.3.13    True        False         False      90m
kube-apiserver                             4.3.13    True        False         False      92m
kube-controller-manager                    4.3.13    True        False         False      93m
kube-scheduler                             4.3.13    True        False         False      93m
machine-api                                4.3.13    True        False         False      95m
machine-config                             4.3.13    True        False         False      94m
marketplace                                4.3.13    True        False         False      37m
monitoring                                 4.3.13    True        False         False      32m
network                                    4.3.13    True        False         False      95m
node-tuning                                4.3.13    True        False         False      38m
openshift-apiserver                        4.3.13    True        False         False      32m
openshift-controller-manager               4.3.13    True        False         False      91m
openshift-samples                          4.3.13    True        False         False      29m
operator-lifecycle-manager                 4.3.13    True        False         False      91m
operator-lifecycle-manager-catalog         4.3.13    True        False         False      91m
operator-lifecycle-manager-packageserver   4.3.13    True        False         False      35m
service-ca                                 4.3.13    True        False         False      95m
service-catalog-apiserver                  4.3.13    True        False         False      91m
service-catalog-controller-manager         4.3.13    True        False         False      90m
storage                                    4.3.13    True        False         False      89m


Note You need to log in before you can comment on or make changes to this bug.