Description of problem: Some pods in Terminating status after deleing projects during Reliability-v2 test long run. Reliability-v2 test: run 15 users concurrently, each user's steps: create 2 project, check all projects under this user, new app, access app, build app in 1 project, scale up app to 2 and scale down to 1 in the 2 projects, check pods in the 2 projects, delete the 2 projects, delete all projects under the user. Version-Release number of selected component (if applicable): 4.10.0-0.nightly-2021-12-23-153012 How reproducible: On 4.10.0-0.nightly-2021aws and azure had problem Steps to Reproduce: 1. Install an aws or azure cluster 2. Run Reliability-v2 for 7 day 3. Monitor pod/project in terminating status. Actual results: After deletion the projects were stuck in terminating status, because there were pods under the projects in terminating status. 'oc delete pod --force' can delete the problem pods and the problem projects were cleared afterwards. But the problem can reproduce some time later. Checking the Terminating pods, some pod's container were 'Terminated' 'Completed'. Each time when the problem happened, only a part of users were impacted. Each time the impacted users could be same or different, seems randomly to me. Expected results: No pods in Terminating state. Additional info: AWS cluster ------------------------------------------------------------------------------- % oc get namespaces | grep Terminating testuser-2-1 Terminating 4h52m testuser-5-0 Terminating 4h54m testuser-6-1 Terminating 2d5h ------------------------------------------------------------------------------- % oc get pods -n testuser-2-1 NAME READY STATUS RESTARTS AGE nodejs-postgresql-persistent-1-89kfn 0/1 Terminating 0 4h56m % oc get pods -n testuser-5-0 NAME READY STATUS RESTARTS AGE django-psql-persistent-1-qwfrp 0/1 Terminating 0 4h55m % oc get pods -n testuser-6-1 NAME READY STATUS RESTARTS AGE rails-pgsql-persistent-1-24b5n 0/1 Terminating 0 2d5h ------------------------------------------------------------------------------- % oc describe pod nodejs-postgresql-persistent-1-89kfn -n testuser-2-1 Name: nodejs-postgresql-persistent-1-89kfn Namespace: testuser-2-1 Priority: 0 Node: ip-10-0-193-55.us-east-2.compute.internal/10.0.193.55 Start Time: Mon, 10 Jan 2022 06:44:58 +0800 Labels: app.openshift.io/runtime=nodejs deployment=nodejs-postgresql-persistent-1 deploymentconfig=nodejs-postgresql-persistent name=nodejs-postgresql-persistent Annotations: k8s.ovn.org/pod-networks: {"default":{"ip_addresses":["10.129.3.163/23"],"mac_address":"0a:58:0a:81:03:a3","gateway_ips":["10.129.2.1"],"ip_address":"10.129.3.163/2... k8s.v1.cni.cncf.io/network-status: [{ "name": "ovn-kubernetes", "interface": "eth0", "ips": [ "10.129.3.163" ], "mac": "0a:58:0a:81:03:a3", "default": true, "dns": {} }] k8s.v1.cni.cncf.io/networks-status: [{ "name": "ovn-kubernetes", "interface": "eth0", "ips": [ "10.129.3.163" ], "mac": "0a:58:0a:81:03:a3", "default": true, "dns": {} }] openshift.io/deployment-config.latest-version: 1 openshift.io/deployment-config.name: nodejs-postgresql-persistent openshift.io/deployment.name: nodejs-postgresql-persistent-1 openshift.io/generated-by: OpenShiftNewApp openshift.io/scc: restricted Status: Terminating (lasts 4h56m) Termination Grace Period: 30s IP: IPs: <none> Controlled By: ReplicationController/nodejs-postgresql-persistent-1 Containers: nodejs-postgresql-persistent: Container ID: cri-o://9f2488b97ddc77788d935b60d055395e8d541246b2358f4a2ea21ab2a311d57b Image: image-registry.openshift-image-registry.svc:5000/testuser-2-1/nodejs-postgresql-persistent@sha256:2f33a41fb9cb4e3002a228b705b3b623aefa8257e1898ceb8b9015f8efdf2820 Image ID: 91595b627ab101950fe2ad35c72f9b64b54821b798bf017875daf00e100c52c6 Port: 8080/TCP Host Port: 0/TCP State: Terminated Reason: Completed Exit Code: 0 Started: Mon, 10 Jan 2022 06:45:05 +0800 Finished: Mon, 10 Jan 2022 06:46:26 +0800 Ready: False Restart Count: 0 Limits: memory: 256Mi Requests: memory: 256Mi Liveness: http-get http://:8080/ delay=30s timeout=3s period=10s #success=1 #failure=3 Readiness: http-get http://:8080/ delay=3s timeout=3s period=10s #success=1 #failure=3 Environment: DATABASE_SERVICE_NAME: postgresql POSTGRESQL_USER: <set to the key 'database-user' in secret 'nodejs-postgresql-persistent'> Optional: false POSTGRESQL_PASSWORD: <set to the key 'database-password' in secret 'nodejs-postgresql-persistent'> Optional: false POSTGRESQL_DATABASE: my_data POSTGRESQL_ADMIN_PASSWORD: <set to the key 'database-admin-password' in secret 'nodejs-postgresql-persistent'> Optional: false Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-x2mzc (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: kube-api-access-x2mzc: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true ConfigMapName: openshift-service-ca.crt ConfigMapOptional: <nil> QoS Class: Burstable Node-Selectors: <none> Tolerations: node.kubernetes.io/memory-pressure:NoSchedule op=Exists node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: <none> ------------------------------------------------------------------------------- % oc describe pod django-psql-persistent-1-qwfrp -n testuser-5-0 Name: django-psql-persistent-1-qwfrp Namespace: testuser-5-0 Priority: 0 Node: ip-10-0-193-55.us-east-2.compute.internal/10.0.193.55 Start Time: Mon, 10 Jan 2022 06:45:43 +0800 Labels: deployment=django-psql-persistent-1 deploymentconfig=django-psql-persistent name=django-psql-persistent Annotations: k8s.ovn.org/pod-networks: {"default":{"ip_addresses":["10.129.3.169/23"],"mac_address":"0a:58:0a:81:03:a9","gateway_ips":["10.129.2.1"],"ip_address":"10.129.3.169/2... k8s.v1.cni.cncf.io/network-status: [{ "name": "ovn-kubernetes", "interface": "eth0", "ips": [ "10.129.3.169" ], "mac": "0a:58:0a:81:03:a9", "default": true, "dns": {} }] k8s.v1.cni.cncf.io/networks-status: [{ "name": "ovn-kubernetes", "interface": "eth0", "ips": [ "10.129.3.169" ], "mac": "0a:58:0a:81:03:a9", "default": true, "dns": {} }] openshift.io/deployment-config.latest-version: 1 openshift.io/deployment-config.name: django-psql-persistent openshift.io/deployment.name: django-psql-persistent-1 openshift.io/generated-by: OpenShiftNewApp openshift.io/scc: restricted Status: Terminating (lasts 4h57m) Termination Grace Period: 30s IP: 10.129.3.169 IPs: IP: 10.129.3.169 Controlled By: ReplicationController/django-psql-persistent-1 Containers: django-psql-persistent: Container ID: cri-o://a8163699e758f3dba4ca4236c957628d1254fa9816726d5a276cfdcf17d4b12b Image: image-registry.openshift-image-registry.svc:5000/testuser-5-0/django-psql-persistent@sha256:317e3edcc2de3055df7f52f96f09292eff898680e6231ffa4a5dbbcf8017abd6 Image ID: ab2169f8ea5a996116f8d0bf31d4eb2ec8d82861071d35ec2779de7be26d4e6d Port: 8080/TCP Host Port: 0/TCP State: Terminated Reason: Completed Exit Code: 0 Started: Mon, 10 Jan 2022 06:46:00 +0800 Finished: Mon, 10 Jan 2022 06:46:45 +0800 Ready: False Restart Count: 0 Limits: memory: 512Mi Requests: memory: 512Mi Liveness: http-get http://:8080/health delay=30s timeout=3s period=10s #success=1 #failure=3 Readiness: http-get http://:8080/health delay=3s timeout=3s period=10s #success=1 #failure=3 Environment: DATABASE_SERVICE_NAME: postgresql DATABASE_ENGINE: postgresql DATABASE_NAME: default DATABASE_USER: <set to the key 'database-user' in secret 'django-psql-persistent'> Optional: false DATABASE_PASSWORD: <set to the key 'database-password' in secret 'django-psql-persistent'> Optional: false APP_CONFIG: DJANGO_SECRET_KEY: <set to the key 'django-secret-key' in secret 'django-psql-persistent'> Optional: false Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fhpw5 (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: kube-api-access-fhpw5: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true ConfigMapName: openshift-service-ca.crt ConfigMapOptional: <nil> QoS Class: Burstable Node-Selectors: <none> Tolerations: node.kubernetes.io/memory-pressure:NoSchedule op=Exists node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: <none> ------------------------------------------------------------------------------- % oc describe pod rails-pgsql-persistent-1-24b5n -n testuser-6-1 Name: rails-pgsql-persistent-1-24b5n Namespace: testuser-6-1 Priority: 0 Node: ip-10-0-144-203.us-east-2.compute.internal/10.0.144.203 Start Time: Sat, 08 Jan 2022 06:11:55 +0800 Labels: deployment=rails-pgsql-persistent-1 deploymentconfig=rails-pgsql-persistent name=rails-pgsql-persistent Annotations: k8s.ovn.org/pod-networks: {"default":{"ip_addresses":["10.128.3.14/23"],"mac_address":"0a:58:0a:80:03:0e","gateway_ips":["10.128.2.1"],"ip_address":"10.128.3.14/23"... k8s.v1.cni.cncf.io/network-status: [{ "name": "ovn-kubernetes", "interface": "eth0", "ips": [ "10.128.3.14" ], "mac": "0a:58:0a:80:03:0e", "default": true, "dns": {} }] k8s.v1.cni.cncf.io/networks-status: [{ "name": "ovn-kubernetes", "interface": "eth0", "ips": [ "10.128.3.14" ], "mac": "0a:58:0a:80:03:0e", "default": true, "dns": {} }] openshift.io/deployment-config.latest-version: 1 openshift.io/deployment-config.name: rails-pgsql-persistent openshift.io/deployment.name: rails-pgsql-persistent-1 openshift.io/generated-by: OpenShiftNewApp openshift.io/scc: restricted Status: Terminating (lasts 2d5h) Termination Grace Period: 30s IP: 10.128.3.14 IPs: IP: 10.128.3.14 Controlled By: ReplicationController/rails-pgsql-persistent-1 Containers: rails-pgsql-persistent: Container ID: cri-o://49f95f337f6b4a3cb7b98fa75861e226f5bc0c3acf6bfb6fbf8fb5e973012973 Image: image-registry.openshift-image-registry.svc:5000/testuser-6-1/rails-pgsql-persistent@sha256:fa6cf3da135d10ebd0ca608632f6bf7a1ff97ee6b47cd8ac189e11b45013c979 Image ID: 69bb7ad95d4a18ba96fe69de7087bcb11433e124f1ffa30ca93cec4805aed6d6 Port: 8080/TCP Host Port: 0/TCP State: Terminated Reason: Error Exit Code: 137 Started: Sat, 08 Jan 2022 06:11:57 +0800 Finished: Sat, 08 Jan 2022 06:14:31 +0800 Ready: False Restart Count: 0 Limits: memory: 512Mi Requests: memory: 512Mi Liveness: http-get http://:8080/articles delay=10s timeout=3s period=10s #success=1 #failure=3 Readiness: http-get http://:8080/articles delay=5s timeout=3s period=10s #success=1 #failure=3 Environment: DATABASE_SERVICE_NAME: postgresql POSTGRESQL_USER: <set to the key 'database-user' in secret 'rails-pgsql-persistent'> Optional: false POSTGRESQL_PASSWORD: <set to the key 'database-password' in secret 'rails-pgsql-persistent'> Optional: false SECRET_KEY_BASE: <set to the key 'keybase' in secret 'rails-pgsql-persistent'> Optional: false POSTGRESQL_DATABASE: root POSTGRESQL_MAX_CONNECTIONS: 100 POSTGRESQL_SHARED_BUFFERS: 12MB APPLICATION_DOMAIN: APPLICATION_USER: <set to the key 'application-user' in secret 'rails-pgsql-persistent'> Optional: false APPLICATION_PASSWORD: <set to the key 'application-password' in secret 'rails-pgsql-persistent'> Optional: false RAILS_ENV: production Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xm5qx (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: kube-api-access-xm5qx: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true ConfigMapName: openshift-service-ca.crt ConfigMapOptional: <nil> QoS Class: Burstable Node-Selectors: <none> Tolerations: node.kubernetes.io/memory-pressure:NoSchedule op=Exists node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: <none> ------------------------------------------------------------------------------- Azure cluster ------------------------------------------------------------------------------- % oc get namespaces | grep Terminating testuser-4-0 Terminating 16h testuser-4-1 Terminating 16h testuser-6-0 Terminating 16h testuser-6-1 Terminating 16h ------------------------------------------------------------------------------- % oc get pods -n testuser-4-0 NAME READY STATUS RESTARTS AGE cakephp-mysql-persistent-1-8kczg 0/1 Terminating 0 16h % oc get pods -n testuser-4-1 NAME READY STATUS RESTARTS AGE postgresql-1-hn6fh 0/1 Terminating 0 16h % oc get pods -n testuser-6-0 NAME READY STATUS RESTARTS AGE rails-pgsql-persistent-1-ttvsb 0/1 Terminating 0 16h rails-pgsql-persistent-2-build 0/1 Terminating 0 16h % oc get pods -n testuser-6-1 NAME READY STATUS RESTARTS AGE nodejs-postgresql-persistent-1-n78gl 0/1 Terminating 0 16h postgresql-1-97bl8 0/1 Terminating 0 16h ------------------------------------------------------------------------------- % oc describe pod cakephp-mysql-persistent-1-8kczg -n testuser-4-0 Name: cakephp-mysql-persistent-1-8kczg Namespace: testuser-4-0 Priority: 0 Node: qili-az-lr-0104-01040218-worker-northcentralus-1/10.0.1.4 Start Time: Sun, 09 Jan 2022 19:30:55 +0800 Labels: deployment=cakephp-mysql-persistent-1 deploymentconfig=cakephp-mysql-persistent name=cakephp-mysql-persistent Annotations: k8s.ovn.org/pod-networks: {"default":{"ip_addresses":["10.131.0.158/23"],"mac_address":"0a:58:0a:83:00:9e","gateway_ips":["10.131.0.1"],"ip_address":"10.131.0.158/2... k8s.v1.cni.cncf.io/network-status: [{ "name": "ovn-kubernetes", "interface": "eth0", "ips": [ "10.131.0.158" ], "mac": "0a:58:0a:83:00:9e", "default": true, "dns": {} }] k8s.v1.cni.cncf.io/networks-status: [{ "name": "ovn-kubernetes", "interface": "eth0", "ips": [ "10.131.0.158" ], "mac": "0a:58:0a:83:00:9e", "default": true, "dns": {} }] openshift.io/deployment-config.latest-version: 1 openshift.io/deployment-config.name: cakephp-mysql-persistent openshift.io/deployment.name: cakephp-mysql-persistent-1 openshift.io/generated-by: OpenShiftNewApp openshift.io/scc: restricted Status: Terminating (lasts 16h) Termination Grace Period: 30s IP: IPs: <none> Controlled By: ReplicationController/cakephp-mysql-persistent-1 Containers: cakephp-mysql-persistent: Container ID: cri-o://4aad8cd996fb4d879e83b098687b30bb0e5aeb2048e41321688bd104502d79ff Image: image-registry.openshift-image-registry.svc:5000/testuser-4-0/cakephp-mysql-persistent@sha256:3b885a2908799465421ebac5da627a7e5cb280ec0168a4c2da3b2edc41a2c8c5 Image ID: d89d38ef7763ab3de03e53bf1f29813fae54626fbbe1caf7917bf58f47c57ecb Port: 8080/TCP Host Port: 0/TCP State: Terminated Reason: Completed Exit Code: 0 Started: Sun, 09 Jan 2022 19:30:59 +0800 Finished: Sun, 09 Jan 2022 19:35:40 +0800 Ready: False Restart Count: 0 Limits: memory: 512Mi Requests: memory: 512Mi Liveness: http-get http://:8080/health.php delay=30s timeout=3s period=60s #success=1 #failure=3 Readiness: http-get http://:8080/health.php delay=3s timeout=3s period=60s #success=1 #failure=3 Environment: DATABASE_SERVICE_NAME: mysql DATABASE_ENGINE: mysql DATABASE_NAME: default DATABASE_USER: <set to the key 'database-user' in secret 'cakephp-mysql-persistent'> Optional: false DATABASE_PASSWORD: <set to the key 'database-password' in secret 'cakephp-mysql-persistent'> Optional: false CAKEPHP_SECRET_TOKEN: <set to the key 'cakephp-secret-token' in secret 'cakephp-mysql-persistent'> Optional: false CAKEPHP_SECURITY_SALT: <set to the key 'cakephp-security-salt' in secret 'cakephp-mysql-persistent'> Optional: false OPCACHE_REVALIDATE_FREQ: 2 Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9nnxj (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: kube-api-access-9nnxj: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true ConfigMapName: openshift-service-ca.crt ConfigMapOptional: <nil> QoS Class: Burstable Node-Selectors: <none> Tolerations: node.kubernetes.io/memory-pressure:NoSchedule op=Exists node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: <none> ------------------------------------------------------------------------------- % oc describe pod postgresql-1-hn6fh -n testuser-4-1 Name: postgresql-1-hn6fh Namespace: testuser-4-1 Priority: 0 Node: qili-az-lr-0104-01040218-worker-northcentralus-1/10.0.1.4 Start Time: Sun, 09 Jan 2022 19:32:11 +0800 Labels: deployment=postgresql-1 deploymentconfig=postgresql name=postgresql Annotations: k8s.ovn.org/pod-networks: {"default":{"ip_addresses":["10.131.0.164/23"],"mac_address":"0a:58:0a:83:00:a4","gateway_ips":["10.131.0.1"],"ip_address":"10.131.0.164/2... k8s.v1.cni.cncf.io/network-status: [{ "name": "ovn-kubernetes", "interface": "eth0", "ips": [ "10.131.0.164" ], "mac": "0a:58:0a:83:00:a4", "default": true, "dns": {} }] k8s.v1.cni.cncf.io/networks-status: [{ "name": "ovn-kubernetes", "interface": "eth0", "ips": [ "10.131.0.164" ], "mac": "0a:58:0a:83:00:a4", "default": true, "dns": {} }] openshift.io/deployment-config.latest-version: 1 openshift.io/deployment-config.name: postgresql openshift.io/deployment.name: postgresql-1 openshift.io/generated-by: OpenShiftNewApp openshift.io/scc: restricted Status: Terminating (lasts 16h) Termination Grace Period: 30s IP: IPs: <none> Controlled By: ReplicationController/postgresql-1 Containers: postgresql: Container ID: cri-o://7255b8445223e9c3e780858f50882197099d7d6fa99894720b1f7bd0dd724f67 Image: image-registry.openshift-image-registry.svc:5000/openshift/postgresql@sha256:906316fde1b69cc21d95904a9983a9bd0ff43b06cde02f9f23f1ef91ec252c63 Image ID: f5624896417d1c44730bceb417c4d75234e8e73307fe3212451371c394c4f29b Port: 5432/TCP Host Port: 0/TCP State: Terminated Reason: Completed Exit Code: 0 Started: Sun, 09 Jan 2022 19:32:31 +0800 Finished: Sun, 09 Jan 2022 19:35:40 +0800 Ready: False Restart Count: 0 Limits: memory: 512Mi Requests: memory: 512Mi Liveness: exec [/usr/libexec/check-container --live] delay=120s timeout=10s period=10s #success=1 #failure=3 Readiness: exec [/usr/libexec/check-container] delay=5s timeout=1s period=10s #success=1 #failure=3 Environment: POSTGRESQL_USER: <set to the key 'database-user' in secret 'django-psql-persistent'> Optional: false POSTGRESQL_PASSWORD: <set to the key 'database-password' in secret 'django-psql-persistent'> Optional: false POSTGRESQL_DATABASE: default Mounts: /var/lib/pgsql/data from postgresql-data (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-j2vs5 (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: postgresql-data: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: postgresql ReadOnly: false kube-api-access-j2vs5: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true ConfigMapName: openshift-service-ca.crt ConfigMapOptional: <nil> QoS Class: Burstable Node-Selectors: <none> Tolerations: node.kubernetes.io/memory-pressure:NoSchedule op=Exists node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: <none> ------------------------------------------------------------------------------- % oc describe pod rails-pgsql-persistent-1-ttvsb -n testuser-6-0 Name: rails-pgsql-persistent-1-ttvsb Namespace: testuser-6-0 Priority: 0 Node: qili-az-lr-0104-01040218-worker-northcentralus-1/10.0.1.4 Start Time: Sun, 09 Jan 2022 19:30:31 +0800 Labels: deployment=rails-pgsql-persistent-1 deploymentconfig=rails-pgsql-persistent name=rails-pgsql-persistent Annotations: k8s.ovn.org/pod-networks: {"default":{"ip_addresses":["10.131.0.154/23"],"mac_address":"0a:58:0a:83:00:9a","gateway_ips":["10.131.0.1"],"ip_address":"10.131.0.154/2... k8s.v1.cni.cncf.io/network-status: [{ "name": "ovn-kubernetes", "interface": "eth0", "ips": [ "10.131.0.154" ], "mac": "0a:58:0a:83:00:9a", "default": true, "dns": {} }] k8s.v1.cni.cncf.io/networks-status: [{ "name": "ovn-kubernetes", "interface": "eth0", "ips": [ "10.131.0.154" ], "mac": "0a:58:0a:83:00:9a", "default": true, "dns": {} }] openshift.io/deployment-config.latest-version: 1 openshift.io/deployment-config.name: rails-pgsql-persistent openshift.io/deployment.name: rails-pgsql-persistent-1 openshift.io/generated-by: OpenShiftNewApp openshift.io/scc: restricted Status: Terminating (lasts 16h) Termination Grace Period: 30s IP: 10.131.0.154 IPs: IP: 10.131.0.154 Controlled By: ReplicationController/rails-pgsql-persistent-1 Containers: rails-pgsql-persistent: Container ID: cri-o://64d530bec6dca38eb7391dcfb8e2d2a214c0d83be975819ad9107b3d4b0f85be Image: image-registry.openshift-image-registry.svc:5000/testuser-6-0/rails-pgsql-persistent@sha256:b1777fa87d654be0d7800f5c0922a2ed8973e49067668136c249b418bbb54804 Image ID: 23ff978e4d5b22ed2bb7540df78680ac37b42b03823936ff271382132f691266 Port: 8080/TCP Host Port: 0/TCP State: Terminated Reason: Error Exit Code: 137 Started: Sun, 09 Jan 2022 19:30:33 +0800 Finished: Sun, 09 Jan 2022 19:36:44 +0800 Ready: False Restart Count: 0 Limits: memory: 512Mi Requests: memory: 512Mi Liveness: http-get http://:8080/articles delay=10s timeout=3s period=10s #success=1 #failure=3 Readiness: http-get http://:8080/articles delay=5s timeout=3s period=10s #success=1 #failure=3 Environment: DATABASE_SERVICE_NAME: postgresql POSTGRESQL_USER: <set to the key 'database-user' in secret 'rails-pgsql-persistent'> Optional: false POSTGRESQL_PASSWORD: <set to the key 'database-password' in secret 'rails-pgsql-persistent'> Optional: false SECRET_KEY_BASE: <set to the key 'keybase' in secret 'rails-pgsql-persistent'> Optional: false POSTGRESQL_DATABASE: root POSTGRESQL_MAX_CONNECTIONS: 100 POSTGRESQL_SHARED_BUFFERS: 12MB APPLICATION_DOMAIN: APPLICATION_USER: <set to the key 'application-user' in secret 'rails-pgsql-persistent'> Optional: false APPLICATION_PASSWORD: <set to the key 'application-password' in secret 'rails-pgsql-persistent'> Optional: false RAILS_ENV: production Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lf9n2 (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: kube-api-access-lf9n2: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true ConfigMapName: openshift-service-ca.crt ConfigMapOptional: <nil> QoS Class: Burstable Node-Selectors: <none> Tolerations: node.kubernetes.io/memory-pressure:NoSchedule op=Exists node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: <none> -------------- %oc describe pod rails-pgsql-persistent-2-build -n testuser-6-0 Name: rails-pgsql-persistent-2-build Namespace: testuser-6-0 Priority: 0 Node: qili-az-lr-0104-01040218-worker-northcentralus-1/10.0.1.4 Start Time: Sun, 09 Jan 2022 19:34:53 +0800 Labels: openshift.io/build.name=rails-pgsql-persistent-2 Annotations: k8s.ovn.org/pod-networks: {"default":{"ip_addresses":["10.131.0.176/23"],"mac_address":"0a:58:0a:83:00:b0","gateway_ips":["10.131.0.1"],"ip_address":"10.131.0.176/2... k8s.v1.cni.cncf.io/network-status: [{ "name": "ovn-kubernetes", "interface": "eth0", "ips": [ "10.131.0.176" ], "mac": "0a:58:0a:83:00:b0", "default": true, "dns": {} }] k8s.v1.cni.cncf.io/networks-status: [{ "name": "ovn-kubernetes", "interface": "eth0", "ips": [ "10.131.0.176" ], "mac": "0a:58:0a:83:00:b0", "default": true, "dns": {} }] openshift.io/build.name: rails-pgsql-persistent-2 openshift.io/scc: privileged Status: Terminating (lasts 16h) Termination Grace Period: 30s IP: IPs: <none> Controlled By: Build/rails-pgsql-persistent-2 Init Containers: git-clone: Container ID: cri-o://643ef3224614381faed67d07285bc31b73216816af75c0e942cbce65d0c22d98 Image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4980169bb178cc8a1ce10ee1379bdefb133051eb57aaa9331af995055f75dbf1 Image ID: bfff7ee8f708a1573d67cabd1e098966c5f5ec9f112b2bf1c4c8747eb90194a2 Port: <none> Host Port: <none> Args: openshift-git-clone --v=0 State: Terminated Reason: Completed Exit Code: 0 Started: Sun, 09 Jan 2022 19:34:56 +0800 Finished: Sun, 09 Jan 2022 19:34:57 +0800 Ready: True Restart Count: 0 Environment: BUILD: {"kind":"Build","apiVersion":"build.openshift.io/v1","metadata":{"name":"rails-pgsql-persistent-2","namespace":"testuser-6-0","uid":"f47bad30-73a7-4961-8fcd-c4ef0de0836b","resourceVersion":"7391194","generation":1,"creationTimestamp":"2022-01-09T11:34:53Z","labels":{"app":"rails-pgsql-persistent","buildconfig":"rails-pgsql-persistent","openshift.io/build-config.name":"rails-pgsql-persistent","openshift.io/build.start-policy":"Serial","template":"rails-pgsql-persistent"},"annotations":{"openshift.io/build-config.name":"rails-pgsql-persistent","openshift.io/build.number":"2"},"ownerReferences":[{"apiVersion":"build.openshift.io/v1","kind":"BuildConfig","name":"rails-pgsql-persistent","uid":"eacc1ae0-38fc-413d-adb0-11a2304349fc","controller":true}],"managedFields":[{"manager":"Go-http-client","operation":"Update","apiVersion":"build.openshift.io/v1","time":"2022-01-09T11:34:53Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:openshift.io/build-config.name":{},"f:openshift.io/build.number":{}},"f:labels":{".":{},"f:app":{},"f:buildconfig":{},"f:openshift.io/build-config.name":{},"f:openshift.io/build.start-policy":{},"f:template":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"eacc1ae0-38fc-413d-adb0-11a2304349fc\"}":{}}},"f:spec":{"f:output":{"f:to":{}},"f:postCommit":{"f:script":{}},"f:serviceAccount":{},"f:source":{"f:git":{".":{},"f:uri":{}},"f:type":{}},"f:strategy":{"f:sourceStrategy":{".":{},"f:env":{},"f:from":{},"f:pullSecret":{}},"f:type":{}},"f:triggeredBy":{}},"f:status":{"f:conditions":{".":{},"k:{\"type\":\"New\"}":{".":{},"f:lastTransitionTime":{},"f:lastUpdateTime":{},"f:status":{},"f:type":{}}},"f:config":{},"f:phase":{}}}}]},"spec":{"serviceAccount":"builder","source":{"type":"Git","git":{"uri":"https://github.com/sclorg/rails-ex.git","httpProxy":"http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@10.0.99.4:3128","httpsProxy":"http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@10.0.99.4:3128","noProxy":".cluster.local,.svc,10.0.0.0/16,10.128.0.0/14,127.0.0.1,169.254.169.254,172.30.0.0/16,api-int.qili-az-lr-0104.qe.azure.devcluster.openshift.com,localhost,test.no-proxy.com"}},"strategy":{"type":"Source","sourceStrategy":{"from":{"kind":"DockerImage","name":"image-registry.openshift-image-registry.svc:5000/openshift/ruby@sha256:e4b3f4d0be0fc8c70a895dae5097035da61c32481e2fb83d6a82a5f95d8f98d3"},"pullSecret":{"name":"builder-dockercfg-5z892"},"env":[{"name":"RUBYGEM_MIRROR"}]}},"output":{"to":{"kind":"DockerImage","name":"image-registry.openshift-image-registry.svc:5000/testuser-6-0/rails-pgsql-persistent:latest"},"pushSecret":{"name":"builder-dockercfg-5z892"}},"resources":{},"postCommit":{"script":"bundle exec rake test"},"nodeSelector":null,"triggeredBy":[{"message":"Manually triggered"}]},"status":{"phase":"New","outputDockerImageReference":"image-registry.openshift-image-registry.svc:5000/testuser-6-0/rails-pgsql-persistent:latest","config":{"kind":"BuildConfig","namespace":"testuser-6-0","name":"rails-pgsql-persistent"},"output":{},"conditions":[{"type":"New","status":"True","lastUpdateTime":"2022-01-09T11:34:53Z","lastTransitionTime":"2022-01-09T11:34:53Z"}]}} LANG: C.utf8 SOURCE_REPOSITORY: https://github.com/sclorg/rails-ex.git SOURCE_URI: https://github.com/sclorg/rails-ex.git ALLOWED_UIDS: 1- DROP_CAPS: KILL,MKNOD,SETGID,SETUID BUILD_REGISTRIES_CONF_PATH: /var/run/configs/openshift.io/build-system/registries.conf BUILD_REGISTRIES_DIR_PATH: /var/run/configs/openshift.io/build-system/registries.d BUILD_SIGNATURE_POLICY_PATH: /var/run/configs/openshift.io/build-system/policy.json BUILD_STORAGE_CONF_PATH: /var/run/configs/openshift.io/build-system/storage.conf BUILD_BLOBCACHE_DIR: /var/cache/blobs HTTP_PROXY: http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@10.0.99.4:3128 HTTPS_PROXY: http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@10.0.99.4:3128 NO_PROXY: .cluster.local,.svc,10.0.0.0/16,10.128.0.0/14,127.0.0.1,169.254.169.254,172.30.0.0/16,api-int.qili-az-lr-0104.qe.azure.devcluster.openshift.com,localhost,test.no-proxy.com Mounts: /tmp/build from buildworkdir (rw) /var/cache/blobs from build-blob-cache (rw) /var/run/configs/openshift.io/build-system from build-system-configs (ro) /var/run/configs/openshift.io/certs from build-ca-bundles (rw) /var/run/configs/openshift.io/pki from build-proxy-ca-bundles (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-gqchv (ro) manage-dockerfile: Container ID: cri-o://c76320f0b0c6cc44660f3c2cd9d66bca246e72e5f7e588fd89340f2487b65455 Image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4980169bb178cc8a1ce10ee1379bdefb133051eb57aaa9331af995055f75dbf1 Image ID: bfff7ee8f708a1573d67cabd1e098966c5f5ec9f112b2bf1c4c8747eb90194a2 Port: <none> Host Port: <none> Args: openshift-manage-dockerfile --v=0 State: Terminated Reason: Completed Exit Code: 0 Started: Sun, 09 Jan 2022 19:34:58 +0800 Finished: Sun, 09 Jan 2022 19:34:58 +0800 Ready: True Restart Count: 0 Environment: BUILD: {"kind":"Build","apiVersion":"build.openshift.io/v1","metadata":{"name":"rails-pgsql-persistent-2","namespace":"testuser-6-0","uid":"f47bad30-73a7-4961-8fcd-c4ef0de0836b","resourceVersion":"7391194","generation":1,"creationTimestamp":"2022-01-09T11:34:53Z","labels":{"app":"rails-pgsql-persistent","buildconfig":"rails-pgsql-persistent","openshift.io/build-config.name":"rails-pgsql-persistent","openshift.io/build.start-policy":"Serial","template":"rails-pgsql-persistent"},"annotations":{"openshift.io/build-config.name":"rails-pgsql-persistent","openshift.io/build.number":"2"},"ownerReferences":[{"apiVersion":"build.openshift.io/v1","kind":"BuildConfig","name":"rails-pgsql-persistent","uid":"eacc1ae0-38fc-413d-adb0-11a2304349fc","controller":true}],"managedFields":[{"manager":"Go-http-client","operation":"Update","apiVersion":"build.openshift.io/v1","time":"2022-01-09T11:34:53Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:openshift.io/build-config.name":{},"f:openshift.io/build.number":{}},"f:labels":{".":{},"f:app":{},"f:buildconfig":{},"f:openshift.io/build-config.name":{},"f:openshift.io/build.start-policy":{},"f:template":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"eacc1ae0-38fc-413d-adb0-11a2304349fc\"}":{}}},"f:spec":{"f:output":{"f:to":{}},"f:postCommit":{"f:script":{}},"f:serviceAccount":{},"f:source":{"f:git":{".":{},"f:uri":{}},"f:type":{}},"f:strategy":{"f:sourceStrategy":{".":{},"f:env":{},"f:from":{},"f:pullSecret":{}},"f:type":{}},"f:triggeredBy":{}},"f:status":{"f:conditions":{".":{},"k:{\"type\":\"New\"}":{".":{},"f:lastTransitionTime":{},"f:lastUpdateTime":{},"f:status":{},"f:type":{}}},"f:config":{},"f:phase":{}}}}]},"spec":{"serviceAccount":"builder","source":{"type":"Git","git":{"uri":"https://github.com/sclorg/rails-ex.git","httpProxy":"http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@10.0.99.4:3128","httpsProxy":"http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@10.0.99.4:3128","noProxy":".cluster.local,.svc,10.0.0.0/16,10.128.0.0/14,127.0.0.1,169.254.169.254,172.30.0.0/16,api-int.qili-az-lr-0104.qe.azure.devcluster.openshift.com,localhost,test.no-proxy.com"}},"strategy":{"type":"Source","sourceStrategy":{"from":{"kind":"DockerImage","name":"image-registry.openshift-image-registry.svc:5000/openshift/ruby@sha256:e4b3f4d0be0fc8c70a895dae5097035da61c32481e2fb83d6a82a5f95d8f98d3"},"pullSecret":{"name":"builder-dockercfg-5z892"},"env":[{"name":"RUBYGEM_MIRROR"}]}},"output":{"to":{"kind":"DockerImage","name":"image-registry.openshift-image-registry.svc:5000/testuser-6-0/rails-pgsql-persistent:latest"},"pushSecret":{"name":"builder-dockercfg-5z892"}},"resources":{},"postCommit":{"script":"bundle exec rake test"},"nodeSelector":null,"triggeredBy":[{"message":"Manually triggered"}]},"status":{"phase":"New","outputDockerImageReference":"image-registry.openshift-image-registry.svc:5000/testuser-6-0/rails-pgsql-persistent:latest","config":{"kind":"BuildConfig","namespace":"testuser-6-0","name":"rails-pgsql-persistent"},"output":{},"conditions":[{"type":"New","status":"True","lastUpdateTime":"2022-01-09T11:34:53Z","lastTransitionTime":"2022-01-09T11:34:53Z"}]}} LANG: C.utf8 SOURCE_REPOSITORY: https://github.com/sclorg/rails-ex.git SOURCE_URI: https://github.com/sclorg/rails-ex.git ALLOWED_UIDS: 1- DROP_CAPS: KILL,MKNOD,SETGID,SETUID BUILD_REGISTRIES_CONF_PATH: /var/run/configs/openshift.io/build-system/registries.conf BUILD_REGISTRIES_DIR_PATH: /var/run/configs/openshift.io/build-system/registries.d BUILD_SIGNATURE_POLICY_PATH: /var/run/configs/openshift.io/build-system/policy.json BUILD_STORAGE_CONF_PATH: /var/run/configs/openshift.io/build-system/storage.conf BUILD_BLOBCACHE_DIR: /var/cache/blobs HTTP_PROXY: http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@10.0.99.4:3128 HTTPS_PROXY: http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@10.0.99.4:3128 NO_PROXY: .cluster.local,.svc,10.0.0.0/16,10.128.0.0/14,127.0.0.1,169.254.169.254,172.30.0.0/16,api-int.qili-az-lr-0104.qe.azure.devcluster.openshift.com,localhost,test.no-proxy.com Mounts: /tmp/build from buildworkdir (rw) /var/cache/blobs from build-blob-cache (rw) /var/run/configs/openshift.io/build-system from build-system-configs (ro) /var/run/configs/openshift.io/certs from build-ca-bundles (rw) /var/run/configs/openshift.io/pki from build-proxy-ca-bundles (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-gqchv (ro) Containers: sti-build: Container ID: cri-o://95da095756dc9944238c410520731ee915ed306e6f1b2268f867e998eec6183b Image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4980169bb178cc8a1ce10ee1379bdefb133051eb57aaa9331af995055f75dbf1 Image ID: bfff7ee8f708a1573d67cabd1e098966c5f5ec9f112b2bf1c4c8747eb90194a2 Port: <none> Host Port: <none> Args: openshift-sti-build --v=0 State: Terminated Reason: Error Message: ommit.author"="Yaakov Selkowitz <yselkowi>" "io.openshift.build.commit.date"="Tue Apr 20 16:10:47 2021 -0400" "io.openshift.build.commit.id"="57f02090043d0285de7e22261e9c407406c01ccf" "io.openshift.build.commit.ref"="master" "io.openshift.build.commit.message"="Fix nokogiri dependencies" "io.openshift.build.source-location"="https://github.com/sclorg/rails-ex.git" [1/3] STEP 3/9: ENV OPENSHIFT_BUILD_NAME="rails-pgsql-persistent-2" OPENSHIFT_BUILD_NAMESPACE="testuser-6-0" OPENSHIFT_BUILD_SOURCE="https://github.com/sclorg/rails-ex.git" OPENSHIFT_BUILD_COMMIT="57f02090043d0285de7e22261e9c407406c01ccf" RUBYGEM_MIRROR="" [1/3] STEP 4/9: USER root [1/3] STEP 5/9: COPY upload/src /tmp/src [1/3] STEP 6/9: RUN chown -R 1001:0 /tmp/src [1/3] STEP 7/9: USER 1001 [1/3] STEP 8/9: RUN /usr/libexec/s2i/assemble ---> Installing application source ... ---> Building your Ruby application from source ... ---> Running 'bundle install --retry 2 --deployment --without development:test' ... The dependency tzinfo-data (>= 0) will be unused by any of the platforms Bundler is installing for. Bundler is installing for ruby but the dependency is only for x86-mingw32, x86-mswin32, x64-mingw32, java. To add those platforms to the bundle, run `bundle lock --add-platform x86-mingw32 x86-mswin32 x64-mingw32 java`. Fetching gem metadata from https://rubygems.org/.......... Fetching rake 13.0.3 Installing rake 13.0.3 Fetching concurrent-ruby 1.1.8 Installing concurrent-ruby 1.1.8 Fetching i18n 1.8.10 Installing i18n 1.8.10 Fetching minitest 5.14.4 Installing minitest 5.14.4 Fetching thread_safe 0.3.6 Installing thread_safe 0.3.6 Fetching tzinfo 1.2.9 Installing tzinfo 1.2.9 Fetching activesupport 5.1.7 Installing activesupport 5.1.7 Fetching builder 3.2.4 Installing builder 3.2.4 Fetching erubi 1.10.0 Installing erubi 1.10.0 Fetching mini_portile2 2.5.0 Installing mini_portile2 2.5.0 Fetching racc 1.5.2 Installing racc 1.5.2 with native extensions Error: received unexpected terminate signal Exit Code: 1 Started: Sun, 09 Jan 2022 19:34:59 +0800 Finished: Sun, 09 Jan 2022 19:35:44 +0800 Ready: False Restart Count: 0 Environment: BUILD: {"kind":"Build","apiVersion":"build.openshift.io/v1","metadata":{"name":"rails-pgsql-persistent-2","namespace":"testuser-6-0","uid":"f47bad30-73a7-4961-8fcd-c4ef0de0836b","resourceVersion":"7391194","generation":1,"creationTimestamp":"2022-01-09T11:34:53Z","labels":{"app":"rails-pgsql-persistent","buildconfig":"rails-pgsql-persistent","openshift.io/build-config.name":"rails-pgsql-persistent","openshift.io/build.start-policy":"Serial","template":"rails-pgsql-persistent"},"annotations":{"openshift.io/build-config.name":"rails-pgsql-persistent","openshift.io/build.number":"2"},"ownerReferences":[{"apiVersion":"build.openshift.io/v1","kind":"BuildConfig","name":"rails-pgsql-persistent","uid":"eacc1ae0-38fc-413d-adb0-11a2304349fc","controller":true}],"managedFields":[{"manager":"Go-http-client","operation":"Update","apiVersion":"build.openshift.io/v1","time":"2022-01-09T11:34:53Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:openshift.io/build-config.name":{},"f:openshift.io/build.number":{}},"f:labels":{".":{},"f:app":{},"f:buildconfig":{},"f:openshift.io/build-config.name":{},"f:openshift.io/build.start-policy":{},"f:template":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"eacc1ae0-38fc-413d-adb0-11a2304349fc\"}":{}}},"f:spec":{"f:output":{"f:to":{}},"f:postCommit":{"f:script":{}},"f:serviceAccount":{},"f:source":{"f:git":{".":{},"f:uri":{}},"f:type":{}},"f:strategy":{"f:sourceStrategy":{".":{},"f:env":{},"f:from":{},"f:pullSecret":{}},"f:type":{}},"f:triggeredBy":{}},"f:status":{"f:conditions":{".":{},"k:{\"type\":\"New\"}":{".":{},"f:lastTransitionTime":{},"f:lastUpdateTime":{},"f:status":{},"f:type":{}}},"f:config":{},"f:phase":{}}}}]},"spec":{"serviceAccount":"builder","source":{"type":"Git","git":{"uri":"https://github.com/sclorg/rails-ex.git","httpProxy":"http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@10.0.99.4:3128","httpsProxy":"http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@10.0.99.4:3128","noProxy":".cluster.local,.svc,10.0.0.0/16,10.128.0.0/14,127.0.0.1,169.254.169.254,172.30.0.0/16,api-int.qili-az-lr-0104.qe.azure.devcluster.openshift.com,localhost,test.no-proxy.com"}},"strategy":{"type":"Source","sourceStrategy":{"from":{"kind":"DockerImage","name":"image-registry.openshift-image-registry.svc:5000/openshift/ruby@sha256:e4b3f4d0be0fc8c70a895dae5097035da61c32481e2fb83d6a82a5f95d8f98d3"},"pullSecret":{"name":"builder-dockercfg-5z892"},"env":[{"name":"RUBYGEM_MIRROR"}]}},"output":{"to":{"kind":"DockerImage","name":"image-registry.openshift-image-registry.svc:5000/testuser-6-0/rails-pgsql-persistent:latest"},"pushSecret":{"name":"builder-dockercfg-5z892"}},"resources":{},"postCommit":{"script":"bundle exec rake test"},"nodeSelector":null,"triggeredBy":[{"message":"Manually triggered"}]},"status":{"phase":"New","outputDockerImageReference":"image-registry.openshift-image-registry.svc:5000/testuser-6-0/rails-pgsql-persistent:latest","config":{"kind":"BuildConfig","namespace":"testuser-6-0","name":"rails-pgsql-persistent"},"output":{},"conditions":[{"type":"New","status":"True","lastUpdateTime":"2022-01-09T11:34:53Z","lastTransitionTime":"2022-01-09T11:34:53Z"}]}} LANG: C.utf8 SOURCE_REPOSITORY: https://github.com/sclorg/rails-ex.git SOURCE_URI: https://github.com/sclorg/rails-ex.git ALLOWED_UIDS: 1- DROP_CAPS: KILL,MKNOD,SETGID,SETUID PUSH_DOCKERCFG_PATH: /var/run/secrets/openshift.io/push PULL_DOCKERCFG_PATH: /var/run/secrets/openshift.io/pull BUILD_REGISTRIES_CONF_PATH: /var/run/configs/openshift.io/build-system/registries.conf BUILD_REGISTRIES_DIR_PATH: /var/run/configs/openshift.io/build-system/registries.d BUILD_SIGNATURE_POLICY_PATH: /var/run/configs/openshift.io/build-system/policy.json BUILD_STORAGE_CONF_PATH: /var/run/configs/openshift.io/build-system/storage.conf BUILD_STORAGE_DRIVER: overlay BUILD_BLOBCACHE_DIR: /var/cache/blobs HTTP_PROXY: http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@10.0.99.4:3128 HTTPS_PROXY: http://proxy-user1:JYgU8qRZV4DY4PXJbxJK@10.0.99.4:3128 NO_PROXY: .cluster.local,.svc,10.0.0.0/16,10.128.0.0/14,127.0.0.1,169.254.169.254,172.30.0.0/16,api-int.qili-az-lr-0104.qe.azure.devcluster.openshift.com,localhost,test.no-proxy.com Mounts: /tmp/build from buildworkdir (rw) /var/cache/blobs from build-blob-cache (rw) /var/lib/containers/cache from buildcachedir (rw) /var/lib/containers/storage from container-storage-root (rw) /var/lib/kubelet/config.json from node-pullsecrets (rw) /var/run/configs/openshift.io/build-system from build-system-configs (ro) /var/run/configs/openshift.io/certs from build-ca-bundles (rw) /var/run/configs/openshift.io/pki from build-proxy-ca-bundles (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-gqchv (ro) /var/run/secrets/openshift.io/pull from builder-dockercfg-5z892-pull (ro) /var/run/secrets/openshift.io/push from builder-dockercfg-5z892-push (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: node-pullsecrets: Type: HostPath (bare host directory volume) Path: /var/lib/kubelet/config.json HostPathType: File buildcachedir: Type: HostPath (bare host directory volume) Path: /var/lib/containers/cache HostPathType: buildworkdir: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: <unset> builder-dockercfg-5z892-push: Type: Secret (a volume populated by a Secret) SecretName: builder-dockercfg-5z892 Optional: false builder-dockercfg-5z892-pull: Type: Secret (a volume populated by a Secret) SecretName: builder-dockercfg-5z892 Optional: false build-system-configs: Type: ConfigMap (a volume populated by a ConfigMap) Name: rails-pgsql-persistent-2-sys-config Optional: false build-ca-bundles: Type: ConfigMap (a volume populated by a ConfigMap) Name: rails-pgsql-persistent-2-ca Optional: false build-proxy-ca-bundles: Type: ConfigMap (a volume populated by a ConfigMap) Name: rails-pgsql-persistent-2-global-ca Optional: false container-storage-root: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: <unset> build-blob-cache: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: <unset> kube-api-access-gqchv: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true ConfigMapName: openshift-service-ca.crt ConfigMapOptional: <nil> QoS Class: BestEffort Node-Selectors: kubernetes.io/os=linux Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: <none> ------------------------------------------------------------------------------- qili@qili-mac bugJan10 % oc describe pod nodejs-postgresql-persistent-1-n78gl -n testuser-6-1 Name: nodejs-postgresql-persistent-1-n78gl Namespace: testuser-6-1 Priority: 0 Node: qili-az-lr-0104-01040218-worker-northcentralus-1/10.0.1.4 Start Time: Sun, 09 Jan 2022 19:35:04 +0800 Labels: app.openshift.io/runtime=nodejs deployment=nodejs-postgresql-persistent-1 deploymentconfig=nodejs-postgresql-persistent name=nodejs-postgresql-persistent Annotations: k8s.ovn.org/pod-networks: {"default":{"ip_addresses":["10.131.0.178/23"],"mac_address":"0a:58:0a:83:00:b2","gateway_ips":["10.131.0.1"],"ip_address":"10.131.0.178/2... k8s.v1.cni.cncf.io/network-status: [{ "name": "ovn-kubernetes", "interface": "eth0", "ips": [ "10.131.0.178" ], "mac": "0a:58:0a:83:00:b2", "default": true, "dns": {} }] k8s.v1.cni.cncf.io/networks-status: [{ "name": "ovn-kubernetes", "interface": "eth0", "ips": [ "10.131.0.178" ], "mac": "0a:58:0a:83:00:b2", "default": true, "dns": {} }] openshift.io/deployment-config.latest-version: 1 openshift.io/deployment-config.name: nodejs-postgresql-persistent openshift.io/deployment.name: nodejs-postgresql-persistent-1 openshift.io/generated-by: OpenShiftNewApp openshift.io/scc: restricted Status: Terminating (lasts 16h) Termination Grace Period: 30s IP: IPs: <none> Controlled By: ReplicationController/nodejs-postgresql-persistent-1 Containers: nodejs-postgresql-persistent: Container ID: cri-o://c11f5c7aa575c7a4a6d9a98ecb3f5896fb05e41538908d177010bdcb48f35972 Image: image-registry.openshift-image-registry.svc:5000/testuser-6-1/nodejs-postgresql-persistent@sha256:dce4d61a0265a11a2219de28fec1a21c5bb1f6979f27ffd74f0f5e960ddc3899 Image ID: 1e0b2e416b33dcb075c54c8d7b2a0a52b04dd02d67c80e8a5ec668fbdb053849 Port: 8080/TCP Host Port: 0/TCP State: Terminated Reason: Completed Exit Code: 0 Started: Sun, 09 Jan 2022 19:35:25 +0800 Finished: Sun, 09 Jan 2022 19:35:43 +0800 Ready: False Restart Count: 0 Limits: memory: 256Mi Requests: memory: 256Mi Liveness: http-get http://:8080/ delay=30s timeout=3s period=10s #success=1 #failure=3 Readiness: http-get http://:8080/ delay=3s timeout=3s period=10s #success=1 #failure=3 Environment: DATABASE_SERVICE_NAME: postgresql POSTGRESQL_USER: <set to the key 'database-user' in secret 'nodejs-postgresql-persistent'> Optional: false POSTGRESQL_PASSWORD: <set to the key 'database-password' in secret 'nodejs-postgresql-persistent'> Optional: false POSTGRESQL_DATABASE: my_data POSTGRESQL_ADMIN_PASSWORD: <set to the key 'database-admin-password' in secret 'nodejs-postgresql-persistent'> Optional: false Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-276ng (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: kube-api-access-276ng: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true ConfigMapName: openshift-service-ca.crt ConfigMapOptional: <nil> QoS Class: Burstable Node-Selectors: <none> Tolerations: node.kubernetes.io/memory-pressure:NoSchedule op=Exists node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: <none> ------------------------------------------------------------------------------- % oc describe pod postgresql-1-97bl8 -n testuser-6-1 Name: postgresql-1-97bl8 Namespace: testuser-6-1 Priority: 0 Node: qili-az-lr-0104-01040218-worker-northcentralus-1/10.0.1.4 Start Time: Sun, 09 Jan 2022 19:30:56 +0800 Labels: deployment=postgresql-1 deploymentconfig=postgresql name=postgresql Annotations: k8s.ovn.org/pod-networks: {"default":{"ip_addresses":["10.131.0.159/23"],"mac_address":"0a:58:0a:83:00:9f","gateway_ips":["10.131.0.1"],"ip_address":"10.131.0.159/2... k8s.v1.cni.cncf.io/network-status: [{ "name": "ovn-kubernetes", "interface": "eth0", "ips": [ "10.131.0.159" ], "mac": "0a:58:0a:83:00:9f", "default": true, "dns": {} }] k8s.v1.cni.cncf.io/networks-status: [{ "name": "ovn-kubernetes", "interface": "eth0", "ips": [ "10.131.0.159" ], "mac": "0a:58:0a:83:00:9f", "default": true, "dns": {} }] openshift.io/deployment-config.latest-version: 1 openshift.io/deployment-config.name: postgresql openshift.io/deployment.name: postgresql-1 openshift.io/generated-by: OpenShiftNewApp openshift.io/scc: restricted Status: Terminating (lasts 16h) Termination Grace Period: 30s IP: IPs: <none> Controlled By: ReplicationController/postgresql-1 Containers: postgresql: Container ID: cri-o://050f1b8970c976e77405ea5633a0ece8000d27a55fbe5f30dc44917d9624768c Image: image-registry.openshift-image-registry.svc:5000/openshift/postgresql@sha256:906316fde1b69cc21d95904a9983a9bd0ff43b06cde02f9f23f1ef91ec252c63 Image ID: f5624896417d1c44730bceb417c4d75234e8e73307fe3212451371c394c4f29b Port: 5432/TCP Host Port: 0/TCP State: Terminated Reason: Completed Exit Code: 0 Started: Sun, 09 Jan 2022 19:31:15 +0800 Finished: Sun, 09 Jan 2022 19:35:44 +0800 Ready: False Restart Count: 0 Limits: memory: 256Mi Requests: memory: 256M -------------------------------------------------------------------------------
*** Bug 2016339 has been marked as a duplicate of this bug. ***
*** Bug 2038507 has been marked as a duplicate of this bug. ***
I have been debugging this to some extent and found the root cause and posted my findings here - https://github.com/kubernetes/kubernetes/issues/96635#issuecomment-1145286675
This is fixed now via - https://github.com/kubernetes/kubernetes/pull/111636
Once Kubernetes rebase lands, this issue should be fixed in OCP.
Depends on - https://github.com/openshift/kubernetes/pull/1348
This should be now fixed via - https://github.com/openshift/kubernetes/pull/1360