We are seeing a number of upgrades where the openshift-apiserver goes degraded but reports MultipleAvailable as the reason. We suspect the underlying cause is the SDN, but this particular BZ report is to break out the MultipleAvailable reason because its actually making it harder to pinpoint the problem. Opened at the request of David.
Sorry for late verifying this bug due to engaged in other ON_QA apiserver bugs and other testings. Tried to verifying it When team do upgrade testing: watched the operator status, didn't find "reason" field with multiple reasons as listed in the PR. Then tried `while true` loop to delete OAS ds, pods and OAS apiservices in parallel, only found ONE reason "NoAPIServerPod", still no multiple ones: "reason": "NoAPIServerPod" Then tried deleting svc: while true; do oc delete svc api -n openshift-apiserver; done Now can get multiple reasons: oc get openshiftapiserver cluster -o json | jq -r '.status.conditions[] | select(.type == "Available")' { "lastTransitionTime": "2019-11-29T08:51:31Z", "message": "apiservice/v1.apps.openshift.io: not available: service/api in \"openshift-apiserver\" is not present\napiservice/v1.authorization.openshift.io: not available: service/api in \"openshift-apiserver\" is not present\napiservice/v1.build.openshift.io: not available: service/api in \"openshift-apiserver\" is not present\napiservice/v1.image.openshift.io: not available: service/api in \"openshift-apiserver\" is not present\napiservice/v1.oauth.openshift.io: not available: service/api in \"openshift-apiserver\" is not present\napiservice/v1.project.openshift.io: not available: service/api in \"openshift-apiserver\" is not present\napiservice/v1.quota.openshift.io: not available: service/api in \"openshift-apiserver\" is not present\napiservice/v1.route.openshift.io: not available: service/api in \"openshift-apiserver\" is not present\napiservice/v1.security.openshift.io: not available: service/api in \"openshift-apiserver\" is not present\napiservice/v1.template.openshift.io: not available: service/api in \"openshift-apiserver\" is not present\napiservice/v1.user.openshift.io: not available: service/api in \"openshift-apiserver\" is not present", "reason": "APIServiceNotAvailable\nAPIServiceNotAvailable\nAPIServiceNotAvailable\nAPIServiceNotAvailable\nAPIServiceNotAvailable\nAPIServiceNotAvailable\nAPIServiceNotAvailable\nAPIServiceNotAvailable\nAPIServiceNotAvailable\nAPIServiceNotAvailable\nAPIServiceNotAvailable", "status": "False", "type": "Available" } In terms of this, the issue can be verified. But if deleting svc and pods in parallel: while true; do oc delete pod -l apiserver -n openshift-apiserver; done # in terminal A while true; do oc delete svc api -n openshift-apiserver; done # in terminal B Only get one reason NoAPIServerPod, no above APIServiceNotAvailable multiple reasons: oc get openshiftapiserver cluster -o json | jq -r '.status.conditions[] | select(.type == "Available")' { "lastTransitionTime": "2019-11-29T08:51:31Z", "message": "no openshift-apiserver daemon pods available on any node.", "reason": "NoAPIServerPod", "status": "False", "type": "Available" } If this is expected, please move back, I'll move to VERIFIED then. Tested version: 4.3.0-0.nightly-2019-11-28-233859. Thanks
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0062