Description of problem: kube-apiserver: Jun 23 12:01:07.819: API was unreachable during disruption for at least 21m51s of 1h19m1s (28%): openshift-apiserver: Jun 23 12:01:07.819: API was unreachable during disruption for at least 21m37s of 1h19m1s (27%): Latest failed job: https://deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-azure-upgrade-4.4-stable-to-4.5-ci/1275367195391561728 There is some evidence of excessive etcd leader changes and some KAS containers crashlooping (which should not lead to disruption). This need to be investigated. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Created attachment 1698480 [details] azure-io
We're asking the following questions to evaluate whether or not this bug warrants blocking an upgrade edge from either the previous X.Y or X.Y.Z. The ultimate goal is to avoid delivering an update which introduces new risk or reduces cluster functionality in any way. Sample answers are provided to give more context and the UpgradeBlocker flag has been added to this bug. It will be removed if the assessment indicates that this should not block upgrade edges. The expectation is that the assignee answers these questions. Who is impacted? If we have to block upgrade edges based on this issue, which edges would need blocking? example: Customers upgrading from 4.y.Z to 4.y+1.z running on GCP with thousands of namespaces, approximately 5% of the subscribed fleet example: All customers upgrading from 4.y.z to 4.y+1.z fail approximately 10% of the time What is the impact? Is it serious enough to warrant blocking edges? example: Up to 2 minute disruption in edge routing example: Up to 90seconds of API downtime example: etcd loses quorum and you have to restore from backup How involved is remediation (even moderately serious impacts might be acceptable if they are easy to mitigate)? example: Issue resolves itself after five minutes example: Admin uses oc to fix things example: Admin must SSH to hosts, restore from backups, or other non standard admin activities Is this a regression (if all previous versions were also vulnerable, updating to the new, vulnerable version does not increase exposure)? example: No, itβs always been like this we just never noticed example: Yes, from 4.y.z to 4.y+1.z Or 4.y.z to 4.y.z+1
Update: etcd team is now working with group-b to better understand how the apiserver failures are calculated. We are focused on ensuring the load balancers are not causing invalid results as calculations are taken external to the cluster. The validity of these values relies on accurate health reporting from load balancers.
Created attachment 1699317 [details] azure apiserver latency 4.4 to 4.5
Created attachment 1699318 [details] azure apiserver latency 4.3 to 4.4
Created attachment 1699319 [details] azure etcd latency 4.3 to 4.4
Created attachment 1699320 [details] azure etcd latency 4.4 to 4.5
As this is going to be a blocker for 4.6, we'll need to prioritize this work in the next sprint or so.
I'm linking https://github.com/openshift/machine-config-operator/issues/1897 , which discusses some possible OSTree-side mitigation strategies.
I think the main idea in https://github.com/openshift/machine-config-operator/issues/1897 is to make the MCO use the same API that Zincati does in FCOS so that we pay the IO cost earlier. So the bulk of the work will be about adapting the MCO rather than RHCOS itself, so re-assigning back to MCO (but obviously work is needed in both; e.g. it'll need at least https://github.com/coreos/rpm-ostree/pull/2158). I left Colin as the assignee in case he wanted to tackle the MCO side.
Status update on this is mostly: Still working on code and tooling to gather more data about whether the proposed changes improve things. - We need to synthesize a "nontrivial" OS update in CI https://github.com/coreos/coreos-assembler/pull/1635 - It took unexpectedly long amount of time to land small "prep work" PRs like https://github.com/openshift/machine-config-operator/pull/1962 - Still waiting on any kind of high level review from the MCO team on https://github.com/openshift/machine-config-operator/pull/1946 - In trying to understand the upgrade tests I stumbled on https://github.com/openshift/origin/pull/25421 for example
VERIFIED with 4.6.0-fc.5 ``` $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.6.0-fc.5 True False 19m Cluster version is 4.6.0-fc.5 $ oc -n openshift-etcd get po NAME READY STATUS RESTARTS AGE etcd-ci-ln-j38n5qt-f76d1-r2ks4-master-0 3/3 Running 0 41m etcd-ci-ln-j38n5qt-f76d1-r2ks4-master-1 3/3 Running 0 26m etcd-ci-ln-j38n5qt-f76d1-r2ks4-master-2 3/3 Running 0 40m etcd-quorum-guard-5c6f86bc54-4h6b8 1/1 Running 0 47m etcd-quorum-guard-5c6f86bc54-5hr9m 1/1 Running 0 47m etcd-quorum-guard-5c6f86bc54-ddvpg 1/1 Running 0 47m installer-2-ci-ln-j38n5qt-f76d1-r2ks4-master-0 0/1 Completed 0 48m installer-2-ci-ln-j38n5qt-f76d1-r2ks4-master-1 0/1 Completed 0 46m installer-2-ci-ln-j38n5qt-f76d1-r2ks4-master-2 0/1 Completed 0 47m installer-3-ci-ln-j38n5qt-f76d1-r2ks4-master-0 0/1 Completed 0 41m installer-3-ci-ln-j38n5qt-f76d1-r2ks4-master-1 0/1 Completed 0 40m installer-3-ci-ln-j38n5qt-f76d1-r2ks4-master-2 0/1 Completed 0 40m revision-pruner-2-ci-ln-j38n5qt-f76d1-r2ks4-master-0 0/1 Completed 0 47m revision-pruner-2-ci-ln-j38n5qt-f76d1-r2ks4-master-1 0/1 Completed 0 46m revision-pruner-2-ci-ln-j38n5qt-f76d1-r2ks4-master-2 0/1 Completed 0 46m revision-pruner-3-ci-ln-j38n5qt-f76d1-r2ks4-master-0 0/1 Completed 0 40m revision-pruner-3-ci-ln-j38n5qt-f76d1-r2ks4-master-1 0/1 Completed 0 25m revision-pruner-3-ci-ln-j38n5qt-f76d1-r2ks4-master-2 0/1 Completed 0 40m $ oc -n openshift-etcd describe pod/etcd-ci-ln-j38n5qt-f76d1-r2ks4-master-0 | grep ionice # See https://etcd.io/docs/v3.4.0/tuning/ for why we use ionice exec ionice -c2 -n0 etcd \ $ oc get nodes NAME STATUS ROLES AGE VERSION ci-ln-j38n5qt-f76d1-r2ks4-master-0 Ready master 51m v1.19.0-rc.2+fc4c489 ci-ln-j38n5qt-f76d1-r2ks4-master-1 Ready master 51m v1.19.0-rc.2+fc4c489 ci-ln-j38n5qt-f76d1-r2ks4-master-2 Ready master 51m v1.19.0-rc.2+fc4c489 ci-ln-j38n5qt-f76d1-r2ks4-worker-b-jg46v Ready worker 41m v1.19.0-rc.2+fc4c489 ci-ln-j38n5qt-f76d1-r2ks4-worker-c-g5rvz Ready worker 41m v1.19.0-rc.2+fc4c489 ci-ln-j38n5qt-f76d1-r2ks4-worker-d-njqvl Ready worker 41m v1.19.0-rc.2+fc4c489 $ oc debug node/ci-ln-j38n5qt-f76d1-r2ks4-master-0 Starting pod/ci-ln-j38n5qt-f76d1-r2ks4-master-0-debug ... To use host binaries, run `chroot /host` Pod IP: 10.0.0.5 If you don't see a command prompt, try pressing enter. sh-4.2# chroot /host sh-4.4# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 128G 0 disk |-sda1 8:1 0 384M 0 part /boot |-sda2 8:2 0 127M 0 part /boot/efi |-sda3 8:3 0 1M 0 part `-sda4 8:4 0 127.5G 0 part `-coreos-luks-root-nocrypt 253:0 0 127.5G 0 dm /sysroot sh-4.4# cat /sys/block/sda/queue/scheduler mq-deadline kyber [bfq] none sh-4.4# sh-4.4# cat /etc/systemd/system/rpm-ostreed.service.d/mco-controlplane-nice.conf # See https://github.com/openshift/machine-config-operator/issues/1897 [Service] Nice=10 IOSchedulingClass=best-effort IOSchedulingPriority=6 sh-4.4# ```
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:4196
Removing UpgradeBlocker from this older bug, to remove it from the suspect queue described in [1]. If you feel like this bug still needs to be a suspect, please add keyword again. [1]: https://github.com/openshift/enhancements/pull/475