Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1462064

Summary:	registry-console pod stuck in terminating state after oc rollout latest
Product:	OpenShift Container Platform	Reporter:	Walid A. <wabouham>
Component:	Node	Assignee:	Derek Carr <decarr>
Status:	CLOSED DUPLICATE	QA Contact:	DeShuai Ma <dma>
Severity:	high	Docs Contact:
Priority:	unspecified
Version:	3.6.0	CC:	aos-bugs, jokerman, mifiedle, mmccomas, sjenning, vlaad, wabouham, wsun
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-06-23 04:09:54 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Walid A. 2017-06-16 04:50:40 UTC

Description of problem:
This is on AWS OCP containerized cluster, 1 master etcd, 1 infra node and two compute nodes.  

registry-console pod gets stuck in terminating state after setting defaultNodeSelector on master-config.yaml followed by oc rollout latest dc/registry-console.  

# oc get pods
NAME                       READY     STATUS        RESTARTS   AGE
docker-registry-2-s8vv4    1/1       Running       1          9h
registry-console-3-7whp0   0/1       Terminating   1          9h
registry-console-4-jprdd   1/1       Running       0          7h
router-4-h4vnk             1/1       Running       0          8h
# 

oc logs registry-console-3-7whp0
json: cannot unmarshal array into Go value of type types.ContainerJSON  

Version-Release number of selected component (if applicable):
openshift v3.6.105
kubernetes v1.6.1+5115d708d7
etcd 3.2.0-rc.1


How reproducible:
Reproducible

Steps to Reproduce:
1. set defaultNodeSelector on master-config.yaml to:

defaultNodeSelector: "region=primary"#g' /etc/origin/master/master-config.yaml


2.  oc annotate namespace default openshift.io/node-selector='region=infra' --overwrite=true

3.  systemctl restart atomic-openshift-master 

4. oc rollout latest dc/registry-console


Actual results:
registry-console-3-7whp0   0/1       Terminating   1          9h
registry-console-4-jprdd   1/1       Running       0          7h

Expected results:
registry-console-3-7whp0  pod should not be stuck in terminating state and be removed, and should not see unmarshalling error in pod log

Additional info:
pod logs, container logs and syslog from master and infra node are provided in next attachements

Comment 3 Walid A. 2017-06-16 05:18:08 UTC

docker logs from infra node:

root@ip-172-31-60-1: ~ # docker ps -a
CONTAINER ID        IMAGE                                                                                                                               COMMAND                  CREATED             STATUS                     PORTS               NAMES
06c00f5225ec        registry.ops.openshift.com/openshift3/registry-console@sha256:7adb2588743571859c35d9a550b5f8d4b8848b07accf1642caef1f9ea28d0421      "/usr/libexec/cockpit"   8 hours ago         Up 8 hours                                     k8s_registry-console_registry-console-4-jprdd_default_c92c6763-5209-11e7-9b13-0263714a0cc4_0
373622a50a42        registry.ops.openshift.com/openshift3/ose-pod:v3.6.105                                                                              "/usr/bin/pod"           8 hours ago         Up 8 hours                                     k8s_POD_registry-console-4-jprdd_default_c92c6763-5209-11e7-9b13-0263714a0cc4_0
3ee835afb6c0        registry.ops.openshift.com/openshift3/ose-haproxy-router@sha256:66b0ed07c48d5d611d2b4f5e509accc1504790aae175aa7cf2038859ea162df0    "/usr/bin/openshift-r"   9 hours ago         Up 9 hours                                     k8s_router_router-4-h4vnk_default_92c90684-5203-11e7-b461-0263714a0cc4_0
aa18dde4a4df        registry.ops.openshift.com/openshift3/ose-pod:v3.6.105                                                                              "/usr/bin/pod"           9 hours ago         Up 9 hours                                     k8s_POD_router-4-h4vnk_default_92c90684-5203-11e7-b461-0263714a0cc4_0
9e3e21635db5        registry.ops.openshift.com/openshift3/ose-docker-registry@sha256:95126e7670bacdc81e9530a0c82703a1e8b78fa4dca57ad70fb41a64ed2c94b5   "/bin/sh -c 'DOCKER_R"   9 hours ago         Up 9 hours                                     k8s_registry_docker-registry-2-s8vv4_default_a74a92d2-51ff-11e7-b461-0263714a0cc4_1
d8571c5affa4        registry.ops.openshift.com/openshift3/registry-console@sha256:7adb2588743571859c35d9a550b5f8d4b8848b07accf1642caef1f9ea28d0421      "/usr/libexec/cockpit"   9 hours ago         Exited (137) 8 hours ago                       k8s_registry-console_registry-console-3-7whp0_default_b919f736-51ff-11e7-b461-0263714a0cc4_1
328fa770daa2        registry.ops.openshift.com/openshift3/ose-pod:v3.6.105                                                                              "/usr/bin/pod"           9 hours ago         Up 9 hours                                     k8s_POD_docker-registry-2-s8vv4_default_a74a92d2-51ff-11e7-b461-0263714a0cc4_1
200bb335e737        registry.ops.openshift.com/openshift3/ose-pod:v3.6.105                                                                              "/usr/bin/pod"           9 hours ago         Exited (0) 8 hours ago                         k8s_POD_registry-console-3-7whp0_default_b919f736-51ff-11e7-b461-0263714a0cc4_1
2479c80133c7        openshift3/node:v3.6.105                                                                                                            "/usr/local/bin/origi"   9 hours ago         Up 9 hours                                     atomic-openshift-node
6d34dd2a649c        openshift3/openvswitch:v3.6.105                                                                                                     "/usr/local/bin/ovs-r"   9 hours ago         Up 9 hours                                     openvswitch
a3e92a042a7a        registry.ops.openshift.com/openshift3/ose-docker-registry@sha256:95126e7670bacdc81e9530a0c82703a1e8b78fa4dca57ad70fb41a64ed2c94b5   "/bin/sh -c 'DOCKER_R"   9 hours ago         Exited (2) 9 hours ago                         k8s_registry_docker-registry-2-s8vv4_default_a74a92d2-51ff-11e7-b461-0263714a0cc4_0
bb3a893f16aa        registry.ops.openshift.com/openshift3/ose-pod:v3.6.105                                                                              "/usr/bin/pod"           9 hours ago         Exited (0) 9 hours ago                         k8s_POD_docker-registry-2-s8vv4_default_a74a92d2-51ff-11e7-b461-0263714a0cc4_0
# 
# docker logs --details d8571c5affa4
 INFO: cockpit-ws: Using certificate: /etc/cockpit/ws-certs.d/0-self-signed.cert
# 
# docker logs --details 200bb335e737
#

Comment 9 Seth Jennings 2017-06-23 04:09:54 UTC

Actually, I am confident enough that it is the same issue as 1460729.

*** This bug has been marked as a duplicate of bug 1460729 ***