Bug 1584725

Summary: [free-stg] label selector for web-console changed between 3.9 and 3.10 - leaves 3.9 pods abandoned
Product: OpenShift Container Platform Reporter: Justin Pierce <jupierce>
Component: Management ConsoleAssignee: Samuel Padgett <spadgett>
Status: CLOSED CURRENTRELEASE QA Contact: Yadan Pei <yapei>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 3.10.0CC: aos-bugs, jokerman, mmccomas, spadgett, yapei
Target Milestone: ---   
Target Release: 3.10.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1588590 (view as bug list) Environment:
Last Closed: 2018-06-11 13:13:13 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1588590    
Attachments:
Description Flags
free-stg namespace listings none

Description Justin Pierce 2018-05-31 13:55:18 UTC
Created attachment 1446306 [details]
free-stg namespace listings

Description of problem:
Upgrading from 3.9.14 to v3.10.0-0.54.0, the 3.9 webconsole pods in openshift-web-console namespace were abandoned. Per spagett, the label selector changed between 3.9 and 3.10 for this deployment. Change appears to have been the addition of 'app: openshift-web-console' in the deployment selector. 

Version-Release number of the following components:
v3.10.0-0.54.0 upgrade

Steps to Reproduce:
1. Upgrade from v3.9.14 to v3.10.0
2. Note that the 3.9 rs/pods in openshift-web-console persist even after the upgrade

Actual results:
3.9 pods stay active after upgrade.

Expected results:
3.9 replicaset should be scaled down/deleted

Comment 1 Samuel Padgett 2018-05-31 14:43:45 UTC
This can happen in a 3.9 -> 3.9.z upgrade, so we'll need to fix in both 3.9 and 3.10.

Comment 4 Yadan Pei 2018-06-07 08:17:16 UTC
1. Setup 3.9.14 cluster
# openshift version
openshift v3.9.14
kubernetes v1.9.1+a0ce1bc657
etcd 3.2.16
2. Upgrade to v3.10.0-0.63.0 using openshift-ansible-3.10.0-0.63.0
# openshift version
openshift v3.10.0-0.63.0
# rpm -q ansible openshift-ansible
ansible-2.4.4.0-1.el7ae.noarch
openshift-ansible-3.10.0-0.63.0.git.0.961c60d.el7.noarch
3. Check resources after upgrade
# oc get all -n openshift-web-console
NAME                              READY     STATUS    RESTARTS   AGE
pod/webconsole-785689b664-dbdv6   1/1       Running   0          31m

NAME                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
service/webconsole   ClusterIP   172.30.207.195   <none>        443/TCP   2h

NAME                         DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/webconsole   1         1         1            1           2h

NAME                                    DESIRED   CURRENT   READY     AGE
replicaset.apps/webconsole-6dc44f7974   0         0         0         2h
replicaset.apps/webconsole-785689b664   1         1         1         31m

# oc get replicaset.apps/webconsole-6dc44f7974 -n openshift-web-console -o=yaml | grep image:
        image: registry.reg-aws.openshift.com:443/openshift3/ose-web-console:v3.9.14

# oc get replicaset.apps/webconsole-785689b664 -n openshift-web-console -o=yaml | grep image:
        image: registry.reg-aws.openshift.com:443/openshift3/ose-web-console:v3.10

3.9 replicasets are scaled down to 0 and no more 3.9 pods