Bug 1810636 - Upon successful deployment of 4.4, shows OOMKilled status for installer-6-master-0
Summary: Upon successful deployment of 4.4, shows OOMKilled status for installer-6-mas...
Keywords:
Status: CLOSED DUPLICATE of bug 1809593
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Node
Version: 4.4
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: Ryan Phillips
QA Contact: Sunil Choudhary
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-03-05 16:09 UTC by rlopez
Modified: 2020-03-12 05:12 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-03-11 13:34:44 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description rlopez 2020-03-05 16:09:09 UTC
Description of problem:

During an attempted deployment of 4.4.0-0.nightly-2020-03-04-204900 , the deployment shows

oc get pods --all-namespaces | grep -iv running | grep -iv complete
openshift-kube-apiserver                                installer-6-master-0.kni4.cloud.lab.eng.bos.redhat.com                0/1     OOMKilled   0          12m

The logs of that container show that it is due to:
I0305 15:56:10.094170       1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert: secrets "user-serving-cert" not found
I0305 15:56:10.294485       1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert-000: secrets "user-serving-cert-000" not found
I0305 15:56:10.494327       1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert-001: secrets "user-serving-cert-001" not found
I0305 15:56:10.693840       1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert-002: secrets "user-serving-cert-002" not found
I0305 15:56:10.894223       1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert-003: secrets "user-serving-cert-003" not found
I0305 15:56:11.094599       1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert-004: secrets "user-serving-cert-004" not found
I0305 15:56:11.294145       1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert-005: secrets "user-serving-cert-005" not found
I0305 15:56:11.494358       1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert-006: secrets "user-serving-cert-006" not found
I0305 15:56:11.694368       1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert-007: secrets "user-serving-cert-007" not found
I0305 15:56:11.893878       1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert-008: secrets "user-serving-cert-008" not found
I0305 15:56:12.094743       1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert-009: secrets "user-serving-cert-009" not found



The full log is here:
$ oc logs -f installer-6-master-0.kni4.cloud.lab.eng.bos.redhat.com -n openshift-kube-apiserver
I0305 15:56:07.489219       1 cmd.go:77] &{<nil> true {false} installer true map[cert-configmaps:0xc000911040 cert-dir:0xc000911220 cert-secrets:0xc000910fa0 configmaps:0xc000910be0 namespace:0xc000910a00 optional-cert-configmaps:0xc000911180 optional-cert-secrets:0xc0009110e0 optional-configmaps:0xc000910d20 optional-secrets:0xc000910c80 pod:0xc000910aa0 pod-manifest-dir:0xc000910e60 resource-dir:0xc000910dc0 revision:0xc000910960 secrets:0xc000910b40 v:0xc0006a00a0] [0xc0006a00a0 0xc000910960 0xc000910a00 0xc000910aa0 0xc000910dc0 0xc000910e60 0xc000910be0 0xc000910d20 0xc000910b40 0xc000910c80 0xc000911220 0xc000911040 0xc000911180 0xc000910fa0 0xc0009110e0] [] map[add-dir-header:0xc0005332c0 alsologtostderr:0xc0005334a0 cert-configmaps:0xc000911040 cert-dir:0xc000911220 cert-secrets:0xc000910fa0 configmaps:0xc000910be0 help:0xc000911cc0 kubeconfig:0xc0009108c0 log-backtrace-at:0xc000533540 log-dir:0xc0005335e0 log-file:0xc000533680 log-file-max-size:0xc000533720 log-flush-frequency:0xc000166b40 logtostderr:0xc0005337c0 namespace:0xc000910a00 optional-cert-configmaps:0xc000911180 optional-cert-secrets:0xc0009110e0 optional-configmaps:0xc000910d20 optional-secrets:0xc000910c80 pod:0xc000910aa0 pod-manifest-dir:0xc000910e60 resource-dir:0xc000910dc0 revision:0xc000910960 secrets:0xc000910b40 skip-headers:0xc000533c20 skip-log-headers:0xc000533cc0 stderrthreshold:0xc0006a0000 timeout-duration:0xc000910f00 v:0xc0006a00a0 vmodule:0xc0006a0140] [0xc0009108c0 0xc000910960 0xc000910a00 0xc000910aa0 0xc000910b40 0xc000910be0 0xc000910c80 0xc000910d20 0xc000910dc0 0xc000910e60 0xc000910f00 0xc000910fa0 0xc000911040 0xc0009110e0 0xc000911180 0xc000911220 0xc0005332c0 0xc0005334a0 0xc000533540 0xc0005335e0 0xc000533680 0xc000533720 0xc000166b40 0xc0005337c0 0xc000533c20 0xc000533cc0 0xc0006a0000 0xc0006a00a0 0xc0006a0140 0xc000911cc0] [0xc0005332c0 0xc0005334a0 0xc000911040 0xc000911220 0xc000910fa0 0xc000910be0 0xc000911cc0 0xc0009108c0 0xc000533540 0xc0005335e0 0xc000533680 0xc000533720 0xc000166b40 0xc0005337c0 0xc000910a00 0xc000911180 0xc0009110e0 0xc000910d20 0xc000910c80 0xc000910aa0 0xc000910e60 0xc000910dc0 0xc000910960 0xc000910b40 0xc000533c20 0xc000533cc0 0xc0006a0000 0xc000910f00 0xc0006a00a0 0xc0006a0140] map[104:0xc000911cc0 118:0xc0006a00a0] [] -1 0 0xc000691f50 true <nil> []}
I0305 15:56:07.489471       1 cmd.go:78] (*installerpod.InstallOptions)(0xc00028aa80)({
 KubeConfig: (string) "",
 KubeClient: (kubernetes.Interface) <nil>,
 Revision: (string) (len=1) "6",
 NodeName: (string) "",
 Namespace: (string) (len=24) "openshift-kube-apiserver",
 PodConfigMapNamePrefix: (string) (len=18) "kube-apiserver-pod",
 SecretNamePrefixes: ([]string) (len=4 cap=4) {
  (string) (len=11) "etcd-client",
  (string) (len=14) "kubelet-client",
  (string) (len=34) "localhost-recovery-serving-certkey",
  (string) (len=31) "localhost-recovery-client-token"
 },
 OptionalSecretNamePrefixes: ([]string) (len=1 cap=1) {
  (string) (len=17) "encryption-config"
 },
 ConfigMapNamePrefixes: ([]string) (len=7 cap=8) {
  (string) (len=18) "kube-apiserver-pod",
  (string) (len=6) "config",
  (string) (len=37) "kube-apiserver-cert-syncer-kubeconfig",
  (string) (len=28) "bound-sa-token-signing-certs",
  (string) (len=15) "etcd-serving-ca",
  (string) (len=18) "kubelet-serving-ca",
  (string) (len=22) "sa-token-signing-certs"
 },
 OptionalConfigMapNamePrefixes: ([]string) (len=3 cap=4) {
  (string) (len=14) "oauth-metadata",
  (string) (len=12) "cloud-config",
  (string) (len=24) "kube-apiserver-server-ca"
 },
 CertSecretNames: ([]string) (len=6 cap=8) {
  (string) (len=17) "aggregator-client",
  (string) (len=30) "localhost-serving-cert-certkey",
  (string) (len=31) "service-network-serving-certkey",
  (string) (len=37) "external-loadbalancer-serving-certkey",
  (string) (len=37) "internal-loadbalancer-serving-certkey",
  (string) (len=33) "bound-service-account-signing-key"
 },
 OptionalCertSecretNamePrefixes: ([]string) (len=11 cap=16) {
  (string) (len=17) "user-serving-cert",
  (string) (len=21) "user-serving-cert-000",
  (string) (len=21) "user-serving-cert-001",
  (string) (len=21) "user-serving-cert-002",
  (string) (len=21) "user-serving-cert-003",
  (string) (len=21) "user-serving-cert-004",
  (string) (len=21) "user-serving-cert-005",
  (string) (len=21) "user-serving-cert-006",
  (string) (len=21) "user-serving-cert-007",
  (string) (len=21) "user-serving-cert-008",
  (string) (len=21) "user-serving-cert-009"
 },
 CertConfigMapNamePrefixes: ([]string) (len=2 cap=2) {
  (string) (len=20) "aggregator-client-ca",
  (string) (len=9) "client-ca"
 },
 OptionalCertConfigMapNamePrefixes: ([]string) (len=1 cap=1) {
  (string) (len=17) "trusted-ca-bundle"
 },
 CertDir: (string) (len=57) "/etc/kubernetes/static-pod-resources/kube-apiserver-certs",
 ResourceDir: (string) (len=36) "/etc/kubernetes/static-pod-resources",
 PodManifestDir: (string) (len=25) "/etc/kubernetes/manifests",
 Timeout: (time.Duration) 2m0s,
 PodMutationFns: ([]installerpod.PodMutationFunc) <nil>
})
I0305 15:56:07.501749       1 cmd.go:254] Creating target resource directory "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6" ...
I0305 15:56:07.501900       1 cmd.go:179] Creating target resource directory "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6" ...
I0305 15:56:07.501913       1 cmd.go:187] Getting secrets ...
I0305 15:56:07.504372       1 copy.go:32] Got secret openshift-kube-apiserver/etcd-client-6
I0305 15:56:07.506390       1 copy.go:32] Got secret openshift-kube-apiserver/kubelet-client-6
I0305 15:56:07.508776       1 copy.go:32] Got secret openshift-kube-apiserver/localhost-recovery-client-token-6
I0305 15:56:07.511016       1 copy.go:32] Got secret openshift-kube-apiserver/localhost-recovery-serving-certkey-6
I0305 15:56:07.513627       1 copy.go:24] Failed to get secret openshift-kube-apiserver/encryption-config-6: secrets "encryption-config-6" not found
I0305 15:56:07.513645       1 cmd.go:200] Getting config maps ...
I0305 15:56:07.515984       1 copy.go:60] Got configMap openshift-kube-apiserver/bound-sa-token-signing-certs-6
I0305 15:56:07.518304       1 copy.go:60] Got configMap openshift-kube-apiserver/config-6
I0305 15:56:07.520195       1 copy.go:60] Got configMap openshift-kube-apiserver/etcd-serving-ca-6
I0305 15:56:07.522795       1 copy.go:60] Got configMap openshift-kube-apiserver/kube-apiserver-cert-syncer-kubeconfig-6
I0305 15:56:07.694281       1 copy.go:60] Got configMap openshift-kube-apiserver/kube-apiserver-pod-6
I0305 15:56:07.894096       1 copy.go:60] Got configMap openshift-kube-apiserver/kubelet-serving-ca-6
I0305 15:56:08.095192       1 copy.go:60] Got configMap openshift-kube-apiserver/sa-token-signing-certs-6
I0305 15:56:08.294500       1 copy.go:52] Failed to get config map openshift-kube-apiserver/cloud-config-6: configmaps "cloud-config-6" not found
I0305 15:56:08.493960       1 copy.go:60] Got configMap openshift-kube-apiserver/kube-apiserver-server-ca-6
I0305 15:56:08.694599       1 copy.go:60] Got configMap openshift-kube-apiserver/oauth-metadata-6
I0305 15:56:08.694630       1 cmd.go:219] Creating directory "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/secrets/etcd-client" ...
I0305 15:56:08.694790       1 cmd.go:225] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/secrets/etcd-client/tls.crt" ...
I0305 15:56:08.694888       1 cmd.go:225] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/secrets/etcd-client/tls.key" ...
I0305 15:56:08.694976       1 cmd.go:219] Creating directory "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/secrets/kubelet-client" ...
I0305 15:56:08.695033       1 cmd.go:225] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/secrets/kubelet-client/tls.key" ...
I0305 15:56:08.695172       1 cmd.go:225] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/secrets/kubelet-client/tls.crt" ...
I0305 15:56:08.695234       1 cmd.go:219] Creating directory "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/secrets/localhost-recovery-client-token" ...
I0305 15:56:08.695292       1 cmd.go:225] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/secrets/localhost-recovery-client-token/service-ca.crt" ...
I0305 15:56:08.695393       1 cmd.go:225] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/secrets/localhost-recovery-client-token/token" ...
I0305 15:56:08.695464       1 cmd.go:225] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/secrets/localhost-recovery-client-token/ca.crt" ...
I0305 15:56:08.695537       1 cmd.go:225] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/secrets/localhost-recovery-client-token/namespace" ...
I0305 15:56:08.695601       1 cmd.go:219] Creating directory "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/secrets/localhost-recovery-serving-certkey" ...
I0305 15:56:08.695667       1 cmd.go:225] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/secrets/localhost-recovery-serving-certkey/tls.crt" ...
I0305 15:56:08.695724       1 cmd.go:225] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/secrets/localhost-recovery-serving-certkey/tls.key" ...
I0305 15:56:08.695785       1 cmd.go:237] Creating directory "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/configmaps/bound-sa-token-signing-certs" ...
I0305 15:56:08.695878       1 cmd.go:242] Writing config file "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/configmaps/bound-sa-token-signing-certs/service-account-001.pub" ...
I0305 15:56:08.695945       1 cmd.go:237] Creating directory "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/configmaps/config" ...
I0305 15:56:08.696003       1 cmd.go:242] Writing config file "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/configmaps/config/config.yaml" ...
I0305 15:56:08.696071       1 cmd.go:237] Creating directory "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/configmaps/etcd-serving-ca" ...
I0305 15:56:08.696139       1 cmd.go:242] Writing config file "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/configmaps/etcd-serving-ca/ca-bundle.crt" ...
I0305 15:56:08.696208       1 cmd.go:237] Creating directory "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/configmaps/kube-apiserver-cert-syncer-kubeconfig" ...
I0305 15:56:08.696267       1 cmd.go:242] Writing config file "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/configmaps/kube-apiserver-cert-syncer-kubeconfig/kubeconfig" ...
I0305 15:56:08.696329       1 cmd.go:237] Creating directory "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/configmaps/kube-apiserver-pod" ...
I0305 15:56:08.696386       1 cmd.go:242] Writing config file "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/configmaps/kube-apiserver-pod/forceRedeploymentReason" ...
I0305 15:56:08.696437       1 cmd.go:242] Writing config file "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/configmaps/kube-apiserver-pod/pod.yaml" ...
I0305 15:56:08.696497       1 cmd.go:242] Writing config file "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/configmaps/kube-apiserver-pod/version" ...
I0305 15:56:08.696558       1 cmd.go:237] Creating directory "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/configmaps/kubelet-serving-ca" ...
I0305 15:56:08.696615       1 cmd.go:242] Writing config file "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/configmaps/kubelet-serving-ca/ca-bundle.crt" ...
I0305 15:56:08.696678       1 cmd.go:237] Creating directory "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/configmaps/sa-token-signing-certs" ...
I0305 15:56:08.696739       1 cmd.go:242] Writing config file "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/configmaps/sa-token-signing-certs/service-account-001.pub" ...
I0305 15:56:08.696798       1 cmd.go:242] Writing config file "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/configmaps/sa-token-signing-certs/service-account-002.pub" ...
I0305 15:56:08.696875       1 cmd.go:237] Creating directory "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/configmaps/kube-apiserver-server-ca" ...
I0305 15:56:08.696933       1 cmd.go:242] Writing config file "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/configmaps/kube-apiserver-server-ca/ca-bundle.crt" ...
I0305 15:56:08.696999       1 cmd.go:237] Creating directory "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/configmaps/oauth-metadata" ...
I0305 15:56:08.697054       1 cmd.go:242] Writing config file "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/configmaps/oauth-metadata/oauthMetadata" ...
I0305 15:56:08.697130       1 cmd.go:179] Creating target resource directory "/etc/kubernetes/static-pod-resources/kube-apiserver-certs" ...
I0305 15:56:08.697146       1 cmd.go:187] Getting secrets ...
I0305 15:56:08.893928       1 copy.go:32] Got secret openshift-kube-apiserver/aggregator-client
I0305 15:56:09.095027       1 copy.go:32] Got secret openshift-kube-apiserver/bound-service-account-signing-key
I0305 15:56:09.293938       1 copy.go:32] Got secret openshift-kube-apiserver/external-loadbalancer-serving-certkey
I0305 15:56:09.493991       1 copy.go:32] Got secret openshift-kube-apiserver/internal-loadbalancer-serving-certkey
I0305 15:56:09.694389       1 copy.go:32] Got secret openshift-kube-apiserver/localhost-serving-cert-certkey
I0305 15:56:09.894306       1 copy.go:32] Got secret openshift-kube-apiserver/service-network-serving-certkey
I0305 15:56:10.094170       1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert: secrets "user-serving-cert" not found
I0305 15:56:10.294485       1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert-000: secrets "user-serving-cert-000" not found
I0305 15:56:10.494327       1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert-001: secrets "user-serving-cert-001" not found
I0305 15:56:10.693840       1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert-002: secrets "user-serving-cert-002" not found
I0305 15:56:10.894223       1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert-003: secrets "user-serving-cert-003" not found
I0305 15:56:11.094599       1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert-004: secrets "user-serving-cert-004" not found
I0305 15:56:11.294145       1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert-005: secrets "user-serving-cert-005" not found
I0305 15:56:11.494358       1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert-006: secrets "user-serving-cert-006" not found
I0305 15:56:11.694368       1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert-007: secrets "user-serving-cert-007" not found
I0305 15:56:11.893878       1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert-008: secrets "user-serving-cert-008" not found
I0305 15:56:12.094743       1 copy.go:24] Failed to get secret openshift-kube-apiserver/user-serving-cert-009: secrets "user-serving-cert-009" not found
I0305 15:56:12.094775       1 cmd.go:200] Getting config maps ...
I0305 15:56:12.294688       1 copy.go:60] Got configMap openshift-kube-apiserver/aggregator-client-ca
I0305 15:56:12.494368       1 copy.go:60] Got configMap openshift-kube-apiserver/client-ca
I0305 15:56:12.705794       1 copy.go:60] Got configMap openshift-kube-apiserver/trusted-ca-bundle
I0305 15:56:12.705840       1 cmd.go:219] Creating directory "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/aggregator-client" ...
I0305 15:56:12.705867       1 cmd.go:225] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/aggregator-client/tls.crt" ...
I0305 15:56:12.706193       1 cmd.go:225] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/aggregator-client/tls.key" ...
I0305 15:56:12.786250       1 cmd.go:219] Creating directory "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/bound-service-account-signing-key" ...
I0305 15:56:12.786278       1 cmd.go:225] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/bound-service-account-signing-key/service-account.key" ...
I0305 15:56:12.786385       1 cmd.go:225] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/bound-service-account-signing-key/service-account.pub" ...
I0305 15:56:12.786521       1 cmd.go:219] Creating directory "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/external-loadbalancer-serving-certkey" ...
I0305 15:56:12.786547       1 cmd.go:225] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/external-loadbalancer-serving-certkey/tls.crt" ...
I0305 15:56:12.786635       1 cmd.go:225] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/external-loadbalancer-serving-certkey/tls.key" ...
I0305 15:56:12.786715       1 cmd.go:219] Creating directory "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey" ...
I0305 15:56:12.786743       1 cmd.go:225] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.crt" ...
I0305 15:56:12.786819       1 cmd.go:225] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.key" ...
I0305 15:56:12.786899       1 cmd.go:219] Creating directory "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/localhost-serving-cert-certkey" ...
I0305 15:56:12.786922       1 cmd.go:225] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/localhost-serving-cert-certkey/tls.crt" ...
I0305 15:56:12.786998       1 cmd.go:225] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/localhost-serving-cert-certkey/tls.key" ...
I0305 15:56:12.787080       1 cmd.go:219] Creating directory "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/service-network-serving-certkey" ...
I0305 15:56:12.787102       1 cmd.go:225] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/service-network-serving-certkey/tls.crt" ...
I0305 15:56:12.787227       1 cmd.go:225] Writing secret manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/service-network-serving-certkey/tls.key" ...
I0305 15:56:12.787314       1 cmd.go:237] Creating directory "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/configmaps/aggregator-client-ca" ...
I0305 15:56:12.787339       1 cmd.go:242] Writing config file "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/configmaps/aggregator-client-ca/ca-bundle.crt" ...
I0305 15:56:12.787429       1 cmd.go:237] Creating directory "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/configmaps/client-ca" ...
I0305 15:56:12.787453       1 cmd.go:242] Writing config file "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/configmaps/client-ca/ca-bundle.crt" ...
I0305 15:56:12.787546       1 cmd.go:237] Creating directory "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/configmaps/trusted-ca-bundle" ...
I0305 15:56:12.787563       1 cmd.go:242] Writing config file "/etc/kubernetes/static-pod-resources/kube-apiserver-certs/configmaps/trusted-ca-bundle/ca-bundle.crt" ...
I0305 15:56:12.787850       1 cmd.go:296] Getting pod configmaps/kube-apiserver-pod-6 -n openshift-kube-apiserver
I0305 15:56:12.894627       1 cmd.go:318] Writing pod manifest "/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6/kube-apiserver-pod.yaml" ...
I0305 15:56:12.894786       1 cmd.go:324] Creating directory for static pod manifest "/etc/kubernetes/manifests" ...
I0305 15:56:12.894806       1 cmd.go:338] Writing static pod manifest "/etc/kubernetes/manifests/kube-apiserver-pod.yaml" ...
{"kind":"Pod","apiVersion":"v1","metadata":{"name":"kube-apiserver","namespace":"openshift-kube-apiserver","creationTimestamp":null,"labels":{"apiserver":"true","app":"openshift-kube-apiserver","revision":"6"},"annotations":{"kubectl.kubernetes.io/default-logs-container":"kube-apiserver"}},"spec":{"volumes":[{"name":"resource-dir","hostPath":{"path":"/etc/kubernetes/static-pod-resources/kube-apiserver-pod-6"}},{"name":"cert-dir","hostPath":{"path":"/etc/kubernetes/static-pod-resources/kube-apiserver-certs"}},{"name":"audit-dir","hostPath":{"path":"/var/log/kube-apiserver"}}],"initContainers":[{"name":"setup","image":"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ea2f3971750c57eae970909f29c1b59cb70f38fa8381cd4ffcefbec6c169120e","command":["/usr/bin/timeout","105","/bin/bash","-ec"],"args":["echo -n \"Fixing audit permissions.\"\nchmod 0700 /var/log/kube-apiserver\necho -n \"Waiting for port :6443 and :6080 to be released.\"\nwhile [ -n \"$(lsof -ni :6443)$(lsof -ni :6080)\" ]; do\n  echo -n \".\"\n  sleep 1\ndone\n"],"resources":{},"volumeMounts":[{"name":"audit-dir","mountPath":"/var/log/kube-apiserver"}],"terminationMessagePolicy":"FallbackToLogsOnError","imagePullPolicy":"IfNotPresent","securityContext":{"privileged":true}}],"containers":[{"name":"kube-apiserver","image":"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ea2f3971750c57eae970909f29c1b59cb70f38fa8381cd4ffcefbec6c169120e","command":["/bin/bash","-ec"],"args":["if [ -f /etc/kubernetes/static-pod-certs/configmaps/trusted-ca-bundle/ca-bundle.crt ]; then\n  echo \"Copying system trust bundle\"\n  cp -f /etc/kubernetes/static-pod-certs/configmaps/trusted-ca-bundle/ca-bundle.crt /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem\nfi\nexec hyperkube kube-apiserver --openshift-config=/etc/kubernetes/static-pod-resources/configmaps/config/config.yaml --advertise-address=${HOST_IP} -v=2"],"ports":[{"containerPort":6443}],"env":[{"name":"POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}},{"name":"STATIC_POD_VERSION","value":"6"},{"name":"HOST_IP","valueFrom":{"fieldRef":{"fieldPath":"status.hostIP"}}}],"resources":{"requests":{"cpu":"150m","memory":"1Gi"}},"volumeMounts":[{"name":"resource-dir","mountPath":"/etc/kubernetes/static-pod-resources"},{"name":"cert-dir","mountPath":"/etc/kubernetes/static-pod-certs"},{"name":"audit-dir","mountPath":"/var/log/kube-apiserver"}],"livenessProbe":{"httpGet":{"path":"healthz","port":6443,"scheme":"HTTPS"},"initialDelaySeconds":45,"timeoutSeconds":10},"readinessProbe":{"httpGet":{"path":"healthz","port":6443,"scheme":"HTTPS"},"initialDelaySeconds":10,"timeoutSeconds":10},"terminationMessagePolicy":"FallbackToLogsOnError","imagePullPolicy":"IfNotPresent","securityContext":{"privileged":true}},{"name":"kube-apiserver-cert-syncer","image":"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2bd27de8a3c69ec44de47df2f3838def7f88b65a61b6036dac3b07a494f74543","command":["cluster-kube-apiserver-operator","cert-syncer"],"args":["--kubeconfig=/etc/kubernetes/static-pod-resources/configmaps/kube-apiserver-cert-syncer-kubeconfig/kubeconfig","--namespace=$(POD_NAMESPACE)","--destination-dir=/etc/kubernetes/static-pod-certs","--tls-server-name-override=localhost-recovery"],"env":[{"name":"POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}}],"resources":{"requests":{"cpu":"10m","memory":"50Mi"}},"volumeMounts":[{"name":"resource-dir","mountPath":"/etc/kubernetes/static-pod-resources"},{"name":"cert-dir","mountPath":"/etc/kubernetes/static-pod-certs"}],"terminationMessagePolicy":"FallbackToLogsOnError","imagePullPolicy":"IfNotPresent"},{"name":"kube-apiserver-cert-regeneration-controller","image":"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2bd27de8a3c69ec44de47df2f3838def7f88b65a61b6036dac3b07a494f74543","command":["cluster-kube-apiserver-operator","cert-regeneration-controller"],"args":["--kubeconfig=/etc/kubernetes/static-pod-resources/configmaps/kube-apiserver-cert-syncer-kubeconfig/kubeconfig","--namespace=$(POD_NAMESPACE)","--tls-server-name=localhost-recovery","-v=2"],"env":[{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}}],"resources":{"requests":{"cpu":"10m","memory":"50Mi"}},"volumeMounts":[{"name":"resource-dir","mountPath":"/etc/kubernetes/static-pod-resources"}],"terminationMessagePolicy":"FallbackToLogsOnError","imagePullPolicy":"IfNotPresent"},{"name":"kube-apiserver-insecure-readyz","image":"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2bd27de8a3c69ec44de47df2f3838def7f88b65a61b6036dac3b07a494f74543","command":["cluster-kube-apiserver-operator","insecure-readyz"],"args":["--insecure-port=6080","--delegate-url=https://localhost:6443/readyz"],"ports":[{"containerPort":6080}],"resources":{"requests":{"cpu":"10m","memory":"50Mi"}},"terminationMessagePolicy":"FallbackToLogsOnError","imagePullPolicy":"IfNotPresent"}],"terminationGracePeriodSeconds":135,"hostNetwork":true,"tolerations":[{"operator":"Exists"}],"priorityClassName":"system-node-critical"},"status":{}}

Comment 1 rlopez 2020-03-05 16:13:47 UTC
To add this OOM killed still gave me a successful install.

Comment 2 rlopez 2020-03-05 16:18:49 UTC
Via slack, someone pointed me to this BZ about installer-6 FYI: https://bugzilla.redhat.com/show_bug.cgi?id=1800609

Comment 3 Stefan Schimanski 2020-03-11 11:45:08 UTC
The installer pods have requests and limits set and are therefore in the Guaranteed QoS class. OOM should not happen. Assigning to node team.

Comment 4 Peter Hunt 2020-03-11 13:34:44 UTC
I am fairly certain this is a dup. Closing as such, please reopen if you disagree

*** This bug has been marked as a duplicate of bug 1809593 ***


Note You need to log in before you can comment on or make changes to this bug.