Bug 1462064 - registry-console pod stuck in terminating state after oc rollout latest
registry-console pod stuck in terminating state after oc rollout latest
Status: CLOSED DUPLICATE of bug 1460729
Product: OpenShift Container Platform
Classification: Red Hat
Component: Pod (Show other bugs)
3.6.0
Unspecified Unspecified
unspecified Severity high
: ---
: ---
Assigned To: Derek Carr
DeShuai Ma
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-16 00:50 EDT by Walid A.
Modified: 2017-08-16 15 EDT (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-06-23 00:09:54 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Walid A. 2017-06-16 00:50:40 EDT
Description of problem:
This is on AWS OCP containerized cluster, 1 master etcd, 1 infra node and two compute nodes.  

registry-console pod gets stuck in terminating state after setting defaultNodeSelector on master-config.yaml followed by oc rollout latest dc/registry-console.  

# oc get pods
NAME                       READY     STATUS        RESTARTS   AGE
docker-registry-2-s8vv4    1/1       Running       1          9h
registry-console-3-7whp0   0/1       Terminating   1          9h
registry-console-4-jprdd   1/1       Running       0          7h
router-4-h4vnk             1/1       Running       0          8h
# 

oc logs registry-console-3-7whp0
json: cannot unmarshal array into Go value of type types.ContainerJSON  

Version-Release number of selected component (if applicable):
openshift v3.6.105
kubernetes v1.6.1+5115d708d7
etcd 3.2.0-rc.1


How reproducible:
Reproducible

Steps to Reproduce:
1. set defaultNodeSelector on master-config.yaml to:

defaultNodeSelector: "region=primary"#g' /etc/origin/master/master-config.yaml


2.  oc annotate namespace default openshift.io/node-selector='region=infra' --overwrite=true

3.  systemctl restart atomic-openshift-master 

4. oc rollout latest dc/registry-console


Actual results:
registry-console-3-7whp0   0/1       Terminating   1          9h
registry-console-4-jprdd   1/1       Running       0          7h

Expected results:
registry-console-3-7whp0  pod should not be stuck in terminating state and be removed, and should not see unmarshalling error in pod log

Additional info:
pod logs, container logs and syslog from master and infra node are provided in next attachements
Comment 3 Walid A. 2017-06-16 01:18:08 EDT
docker logs from infra node:

root@ip-172-31-60-1: ~ # docker ps -a
CONTAINER ID        IMAGE                                                                                                                               COMMAND                  CREATED             STATUS                     PORTS               NAMES
06c00f5225ec        registry.ops.openshift.com/openshift3/registry-console@sha256:7adb2588743571859c35d9a550b5f8d4b8848b07accf1642caef1f9ea28d0421      "/usr/libexec/cockpit"   8 hours ago         Up 8 hours                                     k8s_registry-console_registry-console-4-jprdd_default_c92c6763-5209-11e7-9b13-0263714a0cc4_0
373622a50a42        registry.ops.openshift.com/openshift3/ose-pod:v3.6.105                                                                              "/usr/bin/pod"           8 hours ago         Up 8 hours                                     k8s_POD_registry-console-4-jprdd_default_c92c6763-5209-11e7-9b13-0263714a0cc4_0
3ee835afb6c0        registry.ops.openshift.com/openshift3/ose-haproxy-router@sha256:66b0ed07c48d5d611d2b4f5e509accc1504790aae175aa7cf2038859ea162df0    "/usr/bin/openshift-r"   9 hours ago         Up 9 hours                                     k8s_router_router-4-h4vnk_default_92c90684-5203-11e7-b461-0263714a0cc4_0
aa18dde4a4df        registry.ops.openshift.com/openshift3/ose-pod:v3.6.105                                                                              "/usr/bin/pod"           9 hours ago         Up 9 hours                                     k8s_POD_router-4-h4vnk_default_92c90684-5203-11e7-b461-0263714a0cc4_0
9e3e21635db5        registry.ops.openshift.com/openshift3/ose-docker-registry@sha256:95126e7670bacdc81e9530a0c82703a1e8b78fa4dca57ad70fb41a64ed2c94b5   "/bin/sh -c 'DOCKER_R"   9 hours ago         Up 9 hours                                     k8s_registry_docker-registry-2-s8vv4_default_a74a92d2-51ff-11e7-b461-0263714a0cc4_1
d8571c5affa4        registry.ops.openshift.com/openshift3/registry-console@sha256:7adb2588743571859c35d9a550b5f8d4b8848b07accf1642caef1f9ea28d0421      "/usr/libexec/cockpit"   9 hours ago         Exited (137) 8 hours ago                       k8s_registry-console_registry-console-3-7whp0_default_b919f736-51ff-11e7-b461-0263714a0cc4_1
328fa770daa2        registry.ops.openshift.com/openshift3/ose-pod:v3.6.105                                                                              "/usr/bin/pod"           9 hours ago         Up 9 hours                                     k8s_POD_docker-registry-2-s8vv4_default_a74a92d2-51ff-11e7-b461-0263714a0cc4_1
200bb335e737        registry.ops.openshift.com/openshift3/ose-pod:v3.6.105                                                                              "/usr/bin/pod"           9 hours ago         Exited (0) 8 hours ago                         k8s_POD_registry-console-3-7whp0_default_b919f736-51ff-11e7-b461-0263714a0cc4_1
2479c80133c7        openshift3/node:v3.6.105                                                                                                            "/usr/local/bin/origi"   9 hours ago         Up 9 hours                                     atomic-openshift-node
6d34dd2a649c        openshift3/openvswitch:v3.6.105                                                                                                     "/usr/local/bin/ovs-r"   9 hours ago         Up 9 hours                                     openvswitch
a3e92a042a7a        registry.ops.openshift.com/openshift3/ose-docker-registry@sha256:95126e7670bacdc81e9530a0c82703a1e8b78fa4dca57ad70fb41a64ed2c94b5   "/bin/sh -c 'DOCKER_R"   9 hours ago         Exited (2) 9 hours ago                         k8s_registry_docker-registry-2-s8vv4_default_a74a92d2-51ff-11e7-b461-0263714a0cc4_0
bb3a893f16aa        registry.ops.openshift.com/openshift3/ose-pod:v3.6.105                                                                              "/usr/bin/pod"           9 hours ago         Exited (0) 9 hours ago                         k8s_POD_docker-registry-2-s8vv4_default_a74a92d2-51ff-11e7-b461-0263714a0cc4_0
# 
# docker logs --details d8571c5affa4
 INFO: cockpit-ws: Using certificate: /etc/cockpit/ws-certs.d/0-self-signed.cert
# 
# docker logs --details 200bb335e737
#
Comment 9 Seth Jennings 2017-06-23 00:09:54 EDT
Actually, I am confident enough that it is the same issue as 1460729.

*** This bug has been marked as a duplicate of bug 1460729 ***

Note You need to log in before you can comment on or make changes to this bug.