Bug 1810636
| Summary: | Upon successful deployment of 4.4, shows OOMKilled status for installer-6-master-0 | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | rlopez |
| Component: | Node | Assignee: | Ryan Phillips <rphillips> |
| Status: | CLOSED DUPLICATE | QA Contact: | Sunil Choudhary <schoudha> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 4.4 | CC: | aos-bugs, augol, jokerman, mfojtik, pehunt |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-03-11 13:34:44 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
To add this OOM killed still gave me a successful install. Via slack, someone pointed me to this BZ about installer-6 FYI: https://bugzilla.redhat.com/show_bug.cgi?id=1800609 The installer pods have requests and limits set and are therefore in the Guaranteed QoS class. OOM should not happen. Assigning to node team. I am fairly certain this is a dup. Closing as such, please reopen if you disagree *** This bug has been marked as a duplicate of bug 1809593 *** |
Description of problem: During an attempted deployment of 4.4.0-0.nightly-2020-03-04-204900 , the deployment shows oc get pods --all-namespaces | grep -iv running | grep -iv complete openshift-kube-apiserver installer-6-master-0.kni4.cloud.lab.eng.bos.redhat.com 0/1 OOMKilled 0 12m The logs of that container show that it is due to: I0305 15:56:10.094170 1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert: secrets "user-serving-cert" not found I0305 15:56:10.294485 1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert-000: secrets "user-serving-cert-000" not found I0305 15:56:10.494327 1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert-001: secrets "user-serving-cert-001" not found I0305 15:56:10.693840 1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert-002: secrets "user-serving-cert-002" not found I0305 15:56:10.894223 1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert-003: secrets "user-serving-cert-003" not found I0305 15:56:11.094599 1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert-004: secrets "user-serving-cert-004" not found I0305 15:56:11.294145 1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert-005: secrets "user-serving-cert-005" not found I0305 15:56:11.494358 1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert-006: secrets "user-serving-cert-006" not found I0305 15:56:11.694368 1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert-007: secrets "user-serving-cert-007" not found I0305 15:56:11.893878 1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert-008: secrets "user-serving-cert-008" not found I0305 15:56:12.094743 1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert-009: secrets "user-serving-cert-009" not found The full log is here: $ oc logs -f installer-6-master-0.kni4.cloud.lab.eng.bos.redhat.com -n openshift-kube-apiserver I0305 15:56:07.489219 1 cmd.go:77] &{<nil> true {false} installer true map[cert-configmaps:0xc000911040 cert-dir:0xc000911220 cert-secrets:0xc000910fa0 configmaps:0xc000910be0 namespace:0xc000910a00 optional-cert-configmaps:0xc000911180 optional-cert-secrets:0xc0009110e0 optional-configmaps:0xc000910d20 optional-secrets:0xc000910c80 pod:0xc000910aa0 pod-manifest-dir:0xc000910e60 resource-dir:0xc000910dc0 revision:0xc000910960 secrets:0xc000910b40 v:0xc0006a00a0] [0xc0006a00a0 0xc000910960 0xc000910a00 0xc000910aa0 0xc000910dc0 0xc000910e60 0xc000910be0 0xc000910d20 0xc000910b40 0xc000910c80 0xc000911220 0xc000911040 0xc000911180 0xc000910fa0 0xc0009110e0] [] map[add-dir-header:0xc0005332c0 alsologtostderr:0xc0005334a0 cert-configmaps:0xc000911040 cert-dir:0xc000911220 cert-secrets:0xc000910fa0 configmaps:0xc000910be0 help:0xc000911cc0 kubeconfig:0xc0009108c0 log-backtrace-at:0xc000533540 log-dir:0xc0005335e0 log-file:0xc000533680 log-file-max-size:0xc000533720 log-flush-frequency:0xc000166b40 logtostderr:0xc0005337c0 namespace:0xc000910a00 optional-cert-configmaps:0xc000911180 optional-cert-secrets:0xc0009110e0 optional-configmaps:0xc000910d20 optional-secrets:0xc000910c80 pod:0xc000910aa0 pod-manifest-dir:0xc000910e60 resource-dir:0xc000910dc0 revision:0xc000910960 secrets:0xc000910b40 skip-headers:0xc000533c20 skip-log-headers:0xc000533cc0 stderrthreshold:0xc0006a0000 timeout-duration:0xc000910f00 v:0xc0006a00a0 vmodule:0xc0006a0140] [0xc0009108c0 0xc000910960 0xc000910a00 0xc000910aa0 0xc000910b40 0xc000910be0 0xc000910c80 0xc000910d20 0xc000910dc0 0xc000910e60 0xc000910f00 0xc000910fa0 0xc000911040 0xc0009110e0 0xc000911180 0xc000911220 0xc0005332c0 0xc0005334a0 0xc000533540 0xc0005335e0 0xc000533680 0xc000533720 0xc000166b40 0xc0005337c0 0xc000533c20 0xc000533cc0 0xc0006a0000 0xc0006a00a0 0xc0006a0140 0xc000911cc0] [0xc0005332c0 0xc0005334a0 0xc000911040 0xc000911220 0xc000910fa0 0xc000910be0 0xc000911cc0 0xc0009108c0 0xc000533540 0xc0005335e0 0xc000533680 0xc000533720 0xc000166b40 0xc0005337c0 0xc000910a00 0xc000911180 0xc0009110e0 0xc000910d20 0xc000910c80 0xc000910aa0 0xc000910e60 0xc000910dc0 0xc000910960 0xc000910b40 0xc000533c20 0xc000533cc0 0xc0006a0000 0xc000910f00 0xc0006a00a0 0xc0006a0140] map[104:0xc000911cc0 118:0xc0006a00a0] [] -1 0 0xc000691f50 true <nil> []} I0305 15:56:07.489471 1 cmd.go:78] (*installerpod.InstallOptions)(0xc00028aa80)({ KubeConfig: (string) "", KubeClient: (kubernetes.Interface) <nil>, Revision: (string) (len=1) "6", NodeName: (string) "", Namespace: (string) (len=24) "openshift-kube-apiserver", PodConfigMapNamePrefix: (string) (len=18) "kube-apiserver-pod", SecretNamePrefixes: ([]string) (len=4 cap=4) { (string) (len=11) "etcd-client", (string) (len=14) "kubelet-client", (string) (len=34) "localhost-recovery-serving-certkey", (string) (len=31) "localhost-recovery-client-token" }, OptionalSecretNamePrefixes: ([]string) (len=1 cap=1) { (string) (len=17) "encryption-config" }, ConfigMapNamePrefixes: ([]string) (len=7 cap=8) { (string) (len=18) "kube-apiserver-pod", (string) (len=6) "config", (string) (len=37) "kube-apiserver-cert-syncer-kubeconfig", (string) (len=28) "bound-sa-token-signing-certs", (string) (len=15) "etcd-serving-ca", (string) (len=18) "kubelet-serving-ca", (string) (len=22) "sa-token-signing-certs" }, OptionalConfigMapNamePrefixes: ([]string) (len=3 cap=4) { (string) (len=14) "oauth-metadata", (string) (len=12) "cloud-config", (string) (len=24) "kube-apiserver-server-ca" }, CertSecretNames: ([]string) (len=6 cap=8) { (string) (len=17) "aggregator-client", (string) (len=30) "localhost-serving-cert-certkey", (string) (len=31) "service-network-serving-certkey", (string) (len=37) "external-loadbalancer-serving-certkey", (string) (len=37) "internal-loadbalancer-serving-certkey", (string) (len=33) "bound-service-account-signing-key" }, OptionalCertSecretNamePrefixes: ([]string) (len=11 cap=16) { (string) (len=17) "user-serving-cert", (string) (len=21) "user-serving-cert-000", (string) (len=21) "user-serving-cert-001", (string) (len=21) "user-serving-cert-002", (string) (len=21) "user-serving-cert-003", (string) (len=21) "user-serving-cert-004", (string) (len=21) "user-serving-cert-005", (string) (len=21) "user-serving-cert-006", (string) (len=21) "user-serving-cert-007", (string) (len=21) "user-serving-cert-008", (string) (len=21) "user-serving-cert-009" }, CertConfigMapNamePrefixes: ([]string) (len=2 cap=2) { (string) (len=20) "aggregator-client-ca", (string) (len=9) "client-ca" }, OptionalCertConfigMapNamePrefixes: ([]string) (len=1 cap=1) { (string) (len=17) "trusted-ca-bundle" }, CertDir: (string) (len=57) "/etc/kubernetes/static-pod-resources/kube-apiserver-certs", ResourceDir: (string) (len=36) "/etc/kubernetes/static-pod-resources", PodManifestDir: (string) (len=25) "/etc/kubernetes/manifests", Timeout: (time.Duration) 2m0s, PodMutationFns: ([]installerpod.PodMutationFunc) <nil> }) I0305 15:56:07.501749 1 cmd.go:254] Creating target resource directory "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6" ... I0305 15:56:07.501900 1 cmd.go:179] Creating target resource directory "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6" ... I0305 15:56:07.501913 1 cmd.go:187] Getting secrets ... I0305 15:56:07.504372 1 copy.go:32] Got secret openshift-kube-apiserver/etcd-client-6 I0305 15:56:07.506390 1 copy.go:32] Got secret openshift-kube-apiserver/kubelet-client-6 I0305 15:56:07.508776 1 copy.go:32] Got secret openshift-kube-apiserver/localhost-recovery-client-token-6 I0305 15:56:07.511016 1 copy.go:32] Got secret openshift-kube-apiserver/localhost-recovery-serving-certkey-6 I0305 15:56:07.513627 1 copy.go:24] Failed to get secret openshift-kube-apiserver/encryption-config-6: secrets "encryption-config-6" not found I0305 15:56:07.513645 1 cmd.go:200] Getting config maps ... I0305 15:56:07.515984 1 copy.go:60] Got configMap openshift-kube-apiserver/bound-sa-token-signing-certs-6 I0305 15:56:07.518304 1 copy.go:60] Got configMap openshift-kube-apiserver/config-6 I0305 15:56:07.520195 1 copy.go:60] Got configMap openshift-kube-apiserver/etcd-serving-ca-6 I0305 15:56:07.522795 1 copy.go:60] Got configMap openshift-kube-apiserver/kube-apiserver-cert-syncer-kubeconfig-6 I0305 15:56:07.694281 1 copy.go:60] Got configMap openshift-kube-apiserver/kube-apiserver-pod-6 I0305 15:56:07.894096 1 copy.go:60] Got configMap openshift-kube-apiserver/kubelet-serving-ca-6 I0305 15:56:08.095192 1 copy.go:60] Got configMap openshift-kube-apiserver/sa-token-signing-certs-6 I0305 15:56:08.294500 1 copy.go:52] Failed to get config map openshift-kube-apiserver/cloud-config-6: configmaps "cloud-config-6" not found I0305 15:56:08.493960 1 copy.go:60] Got configMap openshift-kube-apiserver/kube-apiserver-server-ca-6 I0305 15:56:08.694599 1 copy.go:60] Got configMap openshift-kube-apiserver/oauth-metadata-6 I0305 15:56:08.694630 1 cmd.go:219] Creating directory "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/secrets/etcd-client" ... I0305 15:56:08.694790 1 cmd.go:225] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/secrets/etcd-client/tls.crt" ... I0305 15:56:08.694888 1 cmd.go:225] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/secrets/etcd-client/tls.key" ... I0305 15:56:08.694976 1 cmd.go:219] Creating directory "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/secrets/kubelet-client" ... I0305 15:56:08.695033 1 cmd.go:225] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/secrets/kubelet-client/tls.key" ... I0305 15:56:08.695172 1 cmd.go:225] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/secrets/kubelet-client/tls.crt" ... I0305 15:56:08.695234 1 cmd.go:219] Creating directory "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/secrets/localhost-recovery-client-token" ... I0305 15:56:08.695292 1 cmd.go:225] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/secrets/localhost-recovery-client-token/service-ca.crt" ... I0305 15:56:08.695393 1 cmd.go:225] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/secrets/localhost-recovery-client-token/token" ... I0305 15:56:08.695464 1 cmd.go:225] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/secrets/localhost-recovery-client-token/ca.crt" ... I0305 15:56:08.695537 1 cmd.go:225] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/secrets/localhost-recovery-client-token/namespace" ... I0305 15:56:08.695601 1 cmd.go:219] Creating directory "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/secrets/localhost-recovery-serving-certkey" ... I0305 15:56:08.695667 1 cmd.go:225] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/secrets/localhost-recovery-serving-certkey/tls.crt" ... I0305 15:56:08.695724 1 cmd.go:225] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/secrets/localhost-recovery-serving-certkey/tls.key" ... I0305 15:56:08.695785 1 cmd.go:237] Creating directory "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/configmaps/bound-sa-token-signing-certs" ... I0305 15:56:08.695878 1 cmd.go:242] Writing config file "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/configmaps/bound-sa-token-signing-certs/service-account-001.pub" ... I0305 15:56:08.695945 1 cmd.go:237] Creating directory "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/configmaps/config" ... I0305 15:56:08.696003 1 cmd.go:242] Writing config file "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/configmaps/config/config.yaml" ... I0305 15:56:08.696071 1 cmd.go:237] Creating directory "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/configmaps/etcd-serving-ca" ... I0305 15:56:08.696139 1 cmd.go:242] Writing config file "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/configmaps/etcd-serving-ca/ca-bundle.crt" ... I0305 15:56:08.696208 1 cmd.go:237] Creating directory "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/configmaps/kube-apiserver-cert-syncer-kubeconfig" ... I0305 15:56:08.696267 1 cmd.go:242] Writing config file "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/configmaps/kube-apiserver-cert-syncer-kubeconfig/kubeconfig" ... I0305 15:56:08.696329 1 cmd.go:237] Creating directory "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/configmaps/kube-apiserver-pod" ... I0305 15:56:08.696386 1 cmd.go:242] Writing config file "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/configmaps/kube-apiserver-pod/forceRedeploymentReason" ... I0305 15:56:08.696437 1 cmd.go:242] Writing config file "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/configmaps/kube-apiserver-pod/pod.yaml" ... I0305 15:56:08.696497 1 cmd.go:242] Writing config file "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/configmaps/kube-apiserver-pod/version" ... I0305 15:56:08.696558 1 cmd.go:237] Creating directory "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/configmaps/kubelet-serving-ca" ... I0305 15:56:08.696615 1 cmd.go:242] Writing config file "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/configmaps/kubelet-serving-ca/ca-bundle.crt" ... I0305 15:56:08.696678 1 cmd.go:237] Creating directory "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/configmaps/sa-token-signing-certs" ... I0305 15:56:08.696739 1 cmd.go:242] Writing config file "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/configmaps/sa-token-signing-certs/service-account-001.pub" ... I0305 15:56:08.696798 1 cmd.go:242] Writing config file "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/configmaps/sa-token-signing-certs/service-account-002.pub" ... I0305 15:56:08.696875 1 cmd.go:237] Creating directory "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/configmaps/kube-apiserver-server-ca" ... I0305 15:56:08.696933 1 cmd.go:242] Writing config file "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/configmaps/kube-apiserver-server-ca/ca-bundle.crt" ... I0305 15:56:08.696999 1 cmd.go:237] Creating directory "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/configmaps/oauth-metadata" ... I0305 15:56:08.697054 1 cmd.go:242] Writing config file "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/configmaps/oauth-metadata/oauthMetadata" ... I0305 15:56:08.697130 1 cmd.go:179] Creating target resource directory "/etc/kubernetes/static-pod-resources/kube-apiserver-certs" ... I0305 15:56:08.697146 1 cmd.go:187] Getting secrets ... I0305 15:56:08.893928 1 copy.go:32] Got secret openshift-kube-apiserver/aggregator-client I0305 15:56:09.095027 1 copy.go:32] Got secret openshift-kube-apiserver/bound-service-account-signing-key I0305 15:56:09.293938 1 copy.go:32] Got secret openshift-kube-apiserver/external-loadbalancer-serving-certkey I0305 15:56:09.493991 1 copy.go:32] Got secret openshift-kube-apiserver/internal-loadbalancer-serving-certkey I0305 15:56:09.694389 1 copy.go:32] Got secret openshift-kube-apiserver/localhost-serving-cert-certkey I0305 15:56:09.894306 1 copy.go:32] Got secret openshift-kube-apiserver/service-network-serving-certkey I0305 15:56:10.094170 1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert: secrets "user-serving-cert" not found I0305 15:56:10.294485 1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert-000: secrets "user-serving-cert-000" not found I0305 15:56:10.494327 1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert-001: secrets "user-serving-cert-001" not found I0305 15:56:10.693840 1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert-002: secrets "user-serving-cert-002" not found I0305 15:56:10.894223 1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert-003: secrets "user-serving-cert-003" not found I0305 15:56:11.094599 1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert-004: secrets "user-serving-cert-004" not found I0305 15:56:11.294145 1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert-005: secrets "user-serving-cert-005" not found I0305 15:56:11.494358 1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert-006: secrets "user-serving-cert-006" not found I0305 15:56:11.694368 1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert-007: secrets "user-serving-cert-007" not found I0305 15:56:11.893878 1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert-008: secrets "user-serving-cert-008" not found I0305 15:56:12.094743 1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert-009: secrets "user-serving-cert-009" not found I0305 15:56:12.094775 1 cmd.go:200] Getting config maps ... I0305 15:56:12.294688 1 copy.go:60] Got configMap openshift-kube-apiserver/aggregator-client-ca I0305 15:56:12.494368 1 copy.go:60] Got configMap openshift-kube-apiserver/client-ca I0305 15:56:12.705794 1 copy.go:60] Got configMap openshift-kube-apiserver/trusted-ca-bundle I0305 15:56:12.705840 1 cmd.go:219] Creating directory "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/aggregator-client" ... I0305 15:56:12.705867 1 cmd.go:225] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/aggregator-client/tls.crt" ... I0305 15:56:12.706193 1 cmd.go:225] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/aggregator-client/tls.key" ... I0305 15:56:12.786250 1 cmd.go:219] Creating directory "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/bound-service-account-signing-key" ... I0305 15:56:12.786278 1 cmd.go:225] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/bound-service-account-signing-key/service-account.key" ... I0305 15:56:12.786385 1 cmd.go:225] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/bound-service-account-signing-key/service-account.pub" ... I0305 15:56:12.786521 1 cmd.go:219] Creating directory "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/external-loadbalancer-serving-certkey" ... I0305 15:56:12.786547 1 cmd.go:225] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/external-loadbalancer-serving-certkey/tls.crt" ... I0305 15:56:12.786635 1 cmd.go:225] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/external-loadbalancer-serving-certkey/tls.key" ... I0305 15:56:12.786715 1 cmd.go:219] Creating directory "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey" ... I0305 15:56:12.786743 1 cmd.go:225] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.crt" ... I0305 15:56:12.786819 1 cmd.go:225] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.key" ... I0305 15:56:12.786899 1 cmd.go:219] Creating directory "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/localhost-serving-cert-certkey" ... I0305 15:56:12.786922 1 cmd.go:225] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/localhost-serving-cert-certkey/tls.crt" ... I0305 15:56:12.786998 1 cmd.go:225] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/localhost-serving-cert-certkey/tls.key" ... I0305 15:56:12.787080 1 cmd.go:219] Creating directory "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/service-network-serving-certkey" ... I0305 15:56:12.787102 1 cmd.go:225] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/service-network-serving-certkey/tls.crt" ... I0305 15:56:12.787227 1 cmd.go:225] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/service-network-serving-certkey/tls.key" ... I0305 15:56:12.787314 1 cmd.go:237] Creating directory "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/configmaps/aggregator-client-ca" ... I0305 15:56:12.787339 1 cmd.go:242] Writing config file "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/configmaps/aggregator-client-ca/ca-bundle.crt" ... I0305 15:56:12.787429 1 cmd.go:237] Creating directory "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/configmaps/client-ca" ... I0305 15:56:12.787453 1 cmd.go:242] Writing config file "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/configmaps/client-ca/ca-bundle.crt" ... I0305 15:56:12.787546 1 cmd.go:237] Creating directory "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/configmaps/trusted-ca-bundle" ... I0305 15:56:12.787563 1 cmd.go:242] Writing config file "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/configmaps/trusted-ca-bundle/ca-bundle.crt" ... I0305 15:56:12.787850 1 cmd.go:296] Getting pod configmaps/kube-apiserver-pod-6 -n openshift-kube-apiserver I0305 15:56:12.894627 1 cmd.go:318] Writing pod manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/kube-apiserver-pod.yaml" ... I0305 15:56:12.894786 1 cmd.go:324] Creating directory for static pod manifest "/etc/kubernetes/manifests" ... I0305 15:56:12.894806 1 cmd.go:338] Writing static pod manifest "/etc/kubernetes/manifests/kube-apiserver-pod.yaml" ... {"kind":"Pod","apiVersion":"v1","metadata":{"name":"kube-apiserver","namespace":"openshift-kube-apiserver","creationTimestamp":null,"labels":{"apiserver":"true","app":"openshift-kube-apiserver","revision":"6"},"annotations":{"kubectl.kubernetes.io/default-logs-container":"kube-apiserver"}},"spec":{"volumes":[{"name":"resource-dir","hostPath":{"path":"/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6"}},{"name":"cert-dir","hostPath":{"path":"/etc/kubernetes/static-pod-resources/kube-apiserver-certs"}},{"name":"audit-dir","hostPath":{"path":"/var/log/kube-apiserver"}}],"initContainers":[{"name":"setup","image":"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ea2f3971750c57eae970909f29c1b59cb70f38fa8381cd4ffcefbec6c169120e","command":["/usr/bin/timeout","105","/bin/bash","-ec"],"args":["echo -n \"Fixing audit permissions.\"\nchmod 0700 /var/log/kube-apiserver\necho -n \"Waiting for port :6443 and :6080 to be released.\"\nwhile [ -n \"$(lsof -ni :6443)$(lsof -ni :6080)\" ]; do\n echo -n \".\"\n sleep 1\ndone\n"],"resources":{},"volumeMounts":[{"name":"audit-dir","mountPath":"/var/log/kube-apiserver"}],"terminationMessagePolicy":"FallbackToLogsOnError","imagePullPolicy":"IfNotPresent","securityContext":{"privileged":true}}],"containers":[{"name":"kube-apiserver","image":"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ea2f3971750c57eae970909f29c1b59cb70f38fa8381cd4ffcefbec6c169120e","command":["/bin/bash","-ec"],"args":["if [ -f /etc/kubernetes/static-pod-certs/configmaps/trusted-ca-bundle/ca-bundle.crt ]; then\n echo \"Copying system trust bundle\"\n cp -f /etc/kubernetes/static-pod-certs/configmaps/trusted-ca-bundle/ca-bundle.crt /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem\nfi\nexec hyperkube kube-apiserver --openshift-config=/etc/kubernetes/static-pod-resources/configmaps/config/config.yaml --advertise-address=${HOST_IP} -v=2"],"ports":[{"containerPort":6443}],"env":[{"name":"POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}},{"name":"STATIC_POD_VERSION","value":"6"},{"name":"HOST_IP","valueFrom":{"fieldRef":{"fieldPath":"status.hostIP"}}}],"resources":{"requests":{"cpu":"150m","memory":"1Gi"}},"volumeMounts":[{"name":"resource-dir","mountPath":"/etc/kubernetes/static-pod-resources"},{"name":"cert-dir","mountPath":"/etc/kubernetes/static-pod-certs"},{"name":"audit-dir","mountPath":"/var/log/kube-apiserver"}],"livenessProbe":{"httpGet":{"path":"healthz","port":6443,"scheme":"HTTPS"},"initialDelaySeconds":45,"timeoutSeconds":10},"readinessProbe":{"httpGet":{"path":"healthz","port":6443,"scheme":"HTTPS"},"initialDelaySeconds":10,"timeoutSeconds":10},"terminationMessagePolicy":"FallbackToLogsOnError","imagePullPolicy":"IfNotPresent","securityContext":{"privileged":true}},{"name":"kube-apiserver-cert-syncer","image":"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2bd27de8a3c69ec44de47df2f3838def7f88b65a61b6036dac3b07a494f74543","command":["cluster-kube-apiserver-operator","cert-syncer"],"args":["--kubeconfig=/etc/kubernetes/static-pod-resources/configmaps/kube-apiserver-cert-syncer-kubeconfig/kubeconfig","--namespace=$(POD_NAMESPACE)","--destination-dir=/etc/kubernetes/static-pod-certs","--tls-server-name-override=localhost-recovery"],"env":[{"name":"POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}}],"resources":{"requests":{"cpu":"10m","memory":"50Mi"}},"volumeMounts":[{"name":"resource-dir","mountPath":"/etc/kubernetes/static-pod-resources"},{"name":"cert-dir","mountPath":"/etc/kubernetes/static-pod-certs"}],"terminationMessagePolicy":"FallbackToLogsOnError","imagePullPolicy":"IfNotPresent"},{"name":"kube-apiserver-cert-regeneration-controller","image":"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2bd27de8a3c69ec44de47df2f3838def7f88b65a61b6036dac3b07a494f74543","command":["cluster-kube-apiserver-operator","cert-regeneration-controller"],"args":["--kubeconfig=/etc/kubernetes/static-pod-resources/configmaps/kube-apiserver-cert-syncer-kubeconfig/kubeconfig","--namespace=$(POD_NAMESPACE)","--tls-server-name=localhost-recovery","-v=2"],"env":[{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}}],"resources":{"requests":{"cpu":"10m","memory":"50Mi"}},"volumeMounts":[{"name":"resource-dir","mountPath":"/etc/kubernetes/static-pod-resources"}],"terminationMessagePolicy":"FallbackToLogsOnError","imagePullPolicy":"IfNotPresent"},{"name":"kube-apiserver-insecure-readyz","image":"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2bd27de8a3c69ec44de47df2f3838def7f88b65a61b6036dac3b07a494f74543","command":["cluster-kube-apiserver-operator","insecure-readyz"],"args":["--insecure-port=6080","--delegate-url=https://localhost:6443/readyz"],"ports":[{"containerPort":6080}],"resources":{"requests":{"cpu":"10m","memory":"50Mi"}},"terminationMessagePolicy":"FallbackToLogsOnError","imagePullPolicy":"IfNotPresent"}],"terminationGracePeriodSeconds":135,"hostNetwork":true,"tolerations":[{"operator":"Exists"}],"priorityClassName":"system-node-critical"},"status":{}}