1577867 – Some basic components images were not updated after upgrade to targeted version

Bug 1577867 - Some basic components images were not updated after upgrade to targeted version

Summary: Some basic components images were not updated after upgrade to targeted version

Keywords:
Status:	CLOSED INSUFFICIENT_DATA
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Cluster Version Operator
Sub Component:
Version:	3.10.0
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	high
Target Milestone:	---
Target Release:	3.10.0
Assignee:	Scott Dodson
QA Contact:	liujia
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1565545
TreeView+	depends on / blocked

Reported:	2018-05-14 10:07 UTC by liujia
Modified:	2018-05-16 06:34 UTC (History)
CC List:	9 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2018-05-16 06:34:20 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Description liujia 2018-05-14 10:07:42 UTC

Description of problem:
Upgrade ocp v3.10(for both minor version and major version upgrade), after upgrade finished. Some basic images(such as ose-pod/router/registry/deployer) were not updated to targeted version.

For v3.9 to v3.10:
# ansible -i hosts all -m shell -a "docker images"
qe-jliu-r39a-node-registry-router-1.0514-id2.qe.rhcloud.com | SUCCESS | rc=0 >>
REPOSITORY                                                                  TAG                 IMAGE ID            CREATED             SIZE
docker-registry.default.svc:5000/install-test/nodejs-mongodb-example        latest              0f87a0beda37        3 hours ago         510 MB
registry.reg-aws.openshift.com:443/openshift3/ose-node                      v3.10               fdb94ace5758        4 days ago          1.2 GB
registry.reg-aws.openshift.com:443/openshift3/ose-pod                       v3.10               fc62a70157bf        4 days ago          214 MB
registry.reg-aws.openshift.com:443/openshift3/ose-haproxy-router            v3.9.29             c9e3892b5514        5 days ago          1.26 GB
registry.reg-aws.openshift.com:443/openshift3/ose-sti-builder               v3.9.29             1e0fe534e139        5 days ago          1.23 GB
registry.reg-aws.openshift.com:443/openshift3/ose-deployer                  v3.9.29             17067913d670        5 days ago          1.23 GB
registry.reg-aws.openshift.com:443/openshift3/ose-docker-registry           v3.9.29             6c03d7d63b96        5 days ago          435 MB
registry.reg-aws.openshift.com:443/openshift3/ose-pod                       v3.9.29             490f82693c58        5 days ago          214 MB
registry.reg-aws.openshift.com:443/openshift3/ose-template-service-broker   v3.9.29             e017c1d36a84        5 days ago          299 MB
registry.reg-aws.openshift.com:443/openshift3/registry-console              v3.9                652dde20058f        5 days ago          233 MB
registry.reg-aws.openshift.com:443/openshift3/ose-ansible-service-broker    v3.9.29             93df26e4b91d        5 days ago          551 MB
registry.access.redhat.com/rhscl/nodejs-6-rhel7                             <none>              31580b83a625        5 weeks ago         503 MB
registry.access.redhat.com/rhscl/mongodb-32-rhel7                           <none>              f400ee758f60        5 weeks ago         446 MB

qe-jliu-r39a-master-etcd-1.0514-id2.qe.rhcloud.com | SUCCESS | rc=0 >>
REPOSITORY                                                                  TAG                 IMAGE ID            CREATED             SIZE
registry.reg-aws.openshift.com:443/openshift3/ose-control-plane             v3.10               2a64c352cce0        2 days ago          635 MB
registry.reg-aws.openshift.com:443/openshift3/ose-node                      v3.10               fdb94ace5758        4 days ago          1.2 GB
registry.reg-aws.openshift.com:443/openshift3/ose-web-console               v3.10               1e235471d6fc        4 days ago          318 MB
registry.reg-aws.openshift.com:443/openshift3/ose-service-catalog           v3.10               13a1bb653216        4 days ago          310 MB
registry.reg-aws.openshift.com:443/openshift3/ose-template-service-broker   v3.10               9205cac4e53d        4 days ago          284 MB
registry.reg-aws.openshift.com:443/openshift3/ose-ansible-service-broker    v3.10               116764f3d0d2        4 days ago          440 MB
registry.reg-aws.openshift.com:443/openshift3/ose-pod                       v3.10.0-0.38.0      fc62a70157bf        4 days ago          214 MB
registry.reg-aws.openshift.com:443/openshift3/ose-deployer                  v3.9.29             17067913d670        5 days ago          1.23 GB
registry.reg-aws.openshift.com:443/openshift3/ose-web-console               v3.9.29             ae9986489fc7        5 days ago          466 MB
registry.reg-aws.openshift.com:443/openshift3/ose-pod                       v3.9.29             490f82693c58        5 days ago          214 MB
registry.reg-aws.openshift.com:443/openshift3/ose-template-service-broker   v3.9.29             e017c1d36a84        5 days ago          299 MB
registry.reg-aws.openshift.com:443/openshift3/registry-console              v3.9                652dde20058f        5 days ago          233 MB
registry.reg-aws.openshift.com:443/openshift3/ose-service-catalog           v3.9.29             6800c342ce7e        5 days ago          288 MB
registry.access.redhat.com/rhel7/etcd                                       3.2.15              4f35b6516d22        5 weeks ago         256 MB
registry.access.redhat.com/rhel7/etcd                                       latest              4f35b6516d22        5 weeks ago         256 MB

For v3.10.0-0.29.0 to latest:
# ansible -i hosts all -m shell -a "docker images"
qe-jliu-rp10-master-etcd-1.0514-u1i.qe.rhcloud.com | SUCCESS | rc=0 >>
REPOSITORY                                                        TAG                 IMAGE ID            CREATED             SIZE
registry.reg-aws.openshift.com:443/openshift3/ose-control-plane   v3.10               2a64c352cce0        2 days ago          635 MB
registry.reg-aws.openshift.com:443/openshift3/ose-node            v3.10               fdb94ace5758        4 days ago          1.2 GB
registry.reg-aws.openshift.com:443/openshift3/ose-web-console     v3.10               1e235471d6fc        4 days ago          318 MB
registry.reg-aws.openshift.com:443/openshift3/registry-console    v3.10               4afe2070205c        4 days ago          232 MB
registry.reg-aws.openshift.com:443/openshift3/ose-node            v3.10.0-0.29.0      1c60f251970e        2 weeks ago         1.83 GB
registry.reg-aws.openshift.com:443/openshift3/ose-deployer        v3.10.0-0.29.0      ca787a3c6f62        2 weeks ago         1.62 GB
registry.reg-aws.openshift.com:443/openshift3/ose-web-console     v3.10.0-0.29.0      47aa15dcc21f        2 weeks ago         470 MB
registry.reg-aws.openshift.com:443/openshift3/ose-control-plane   v3.10.0-0.29.0      2fde04e76b11        2 weeks ago         1.62 GB
registry.reg-aws.openshift.com:443/openshift3/ose-pod             v3.10.0-0.29.0      3cc11b1d60be        2 weeks ago         214 MB
registry.access.redhat.com/rhel7/etcd                             3.2.15              4f35b6516d22        5 weeks ago         256 MB

qe-jliu-rp10-node-registry-router-1.0514-u1i.qe.rhcloud.com | SUCCESS | rc=0 >>
REPOSITORY                                                          TAG                 IMAGE ID            CREATED             SIZE
registry.reg-aws.openshift.com:443/openshift3/ose-node              v3.10               fdb94ace5758        4 days ago          1.2 GB
registry.reg-aws.openshift.com:443/openshift3/ose-node              v3.10.0-0.29.0      1c60f251970e        2 weeks ago         1.83 GB
registry.reg-aws.openshift.com:443/openshift3/ose-haproxy-router    v3.10.0-0.29.0      fcf3839872fa        2 weeks ago         1.64 GB
registry.reg-aws.openshift.com:443/openshift3/ose-deployer          v3.10.0-0.29.0      ca787a3c6f62        2 weeks ago         1.62 GB
registry.reg-aws.openshift.com:443/openshift3/ose-docker-registry   v3.10.0-0.29.0      11b3c9ece026        2 weeks ago         436 MB
registry.reg-aws.openshift.com:443/openshift3/ose-pod               v3.10.0-0.29.0      3cc11b1d60be        2 weeks ago         214 MB


Take docker-registry for example, image was updated in dc, but seems not re-deploy.

[root@qe-jliu-rp10-master-etcd-1 ~]# oc get pod
NAME                       READY     STATUS    RESTARTS   AGE
docker-registry-1-k5tsv    1/1       Running   0          28m
registry-console-1-5kzrr   1/1       Running   3          1h
router-1-57dtl             1/1       Running   0          28m
[root@qe-jliu-rp10-master-etcd-1 ~]# oc get dc
NAME               REVISION   DESIRED   CURRENT   TRIGGERED BY
docker-registry    2          1         0         config
registry-console   1          1         1         config
router             2          1         0         config
[root@qe-jliu-rp10-master-etcd-1 ~]# 
[root@qe-jliu-rp10-master-etcd-1 ~]# 
[root@qe-jliu-rp10-master-etcd-1 ~]# oc get dc docker-registry -o yaml|grep image
        image: registry.reg-aws.openshift.com:443/openshift3/ose-docker-registry:v3.10
        imagePullPolicy: IfNotPresent



Version-Release number of the following components:
openshift-ansible-3.10.0-0.41.0.git.0.88119e4.el7.noarch

How reproducible:
always

Steps to Reproduce:
1. Some basic images(such as ose-pod/router/registry/deployer) were not updated to targeted version.
2.
3.

Actual results:
Please include the entire output from the last TASK line through the end of output if an error is generated

Expected results:
Images should be updated during upgrade.

Additional info:
Please attach logs from ansible-playbook with the -vvv flag

Comment 1 liujia 2018-05-14 10:08:57 UTC

Block other teams to test openshift features realted function after upgrade.

Comment 2 Michael Gugino 2018-05-15 17:20:26 UTC

Please include inventory and -vvv ansible-playbook output.

First indication is that openshift_image_tag was set to 3.9 during upgrade to 3.10, which may have affected some components.

Comment 3 liujia 2018-05-16 06:34:20 UTC

(In reply to Michael Gugino from comment #2)
> Please include inventory and -vvv ansible-playbook output.
> 
> First indication is that openshift_image_tag was set to 3.9 during upgrade
> to 3.10, which may have affected some components.

All upgrade logs have been removed now. But I remembered no openshift_image_tag was set.

For scenario v3.9-v3.10 upgrade, because it was a rpm installed env(install log) so I will not add openshift_image_tag for it at all. And from "docker images" info, all images' tag was v3.10, but not an exact version except especial one openshift3/ose-pod.

qe-jliu-r39a-master-etcd-1.0514-id2.qe.rhcloud.com | SUCCESS | rc=0 >>
REPOSITORY                                                                  TAG                 IMAGE ID            CREATED             SIZE
registry.reg-aws.openshift.com:443/openshift3/ose-control-plane             v3.10               2a64c352cce0        2 days ago          635 MB
registry.reg-aws.openshift.com:443/openshift3/ose-node                      v3.10               fdb94ace5758        4 days ago          1.2 GB
registry.reg-aws.openshift.com:443/openshift3/ose-web-console               v3.10               1e235471d6fc        4 days ago          318 MB
registry.reg-aws.openshift.com:443/openshift3/ose-service-catalog           v3.10               13a1bb653216        4 days ago          310 MB
registry.reg-aws.openshift.com:443/openshift3/ose-template-service-broker   v3.10               9205cac4e53d        4 days ago          284 MB
registry.reg-aws.openshift.com:443/openshift3/ose-ansible-service-broker    v3.10               116764f3d0d2        4 days ago          440 MB
registry.reg-aws.openshift.com:443/openshift3/ose-pod                       v3.10.0-0.38.0      fc62a70157bf        4 days ago          214 MB
registry.reg-aws.openshift.com:443/openshift3/ose-deployer                  v3.9.29             17067913d670        5 days ago          1.23 GB
registry.reg-aws.openshift.com:443/openshift3/ose-web-console               v3.9.29             ae9986489fc7        5 days ago          466 MB
registry.reg-aws.openshift.com:443/openshift3/ose-pod                       v3.9.29             490f82693c58        5 days ago          214 MB
registry.reg-aws.openshift.com:443/openshift3/ose-template-service-broker   v3.9.29             e017c1d36a84        5 days ago          299 MB
registry.reg-aws.openshift.com:443/openshift3/registry-console              v3.9                652dde20058f        5 days ago          233 MB
registry.reg-aws.openshift.com:443/openshift3/ose-service-catalog           v3.9.29             6800c342ce7e        5 days ago          288 MB
registry.access.redhat.com/rhel7/etcd                                       3.2.15              4f35b6516d22        5 weeks ago         256 MB
registry.access.redhat.com/rhel7/etcd                                       latest              4f35b6516d22        5 weeks ago         256 MB

For v3.10.0-0.29.0 to latest upgrade, take web_console images for example, if openshift_image_tag was set, then the image tag should not be v3.10. Because before upgrade, it was v3.10.0-0.29.0(installed with openshift_image_tag specified)

qe-jliu-rp10-master-etcd-1.0514-u1i.qe.rhcloud.com | SUCCESS | rc=0 >>
REPOSITORY                                                        TAG                 IMAGE ID            CREATED             SIZE
registry.reg-aws.openshift.com:443/openshift3/ose-control-plane   v3.10               2a64c352cce0        2 days ago          635 MB
registry.reg-aws.openshift.com:443/openshift3/ose-node            v3.10               fdb94ace5758        4 days ago          1.2 GB
registry.reg-aws.openshift.com:443/openshift3/ose-web-console     v3.10               1e235471d6fc        4 days ago          318 MB
registry.reg-aws.openshift.com:443/openshift3/registry-console    v3.10               4afe2070205c        4 days ago          232 MB
registry.reg-aws.openshift.com:443/openshift3/ose-node            v3.10.0-0.29.0      1c60f251970e        2 weeks ago         1.83 GB
registry.reg-aws.openshift.com:443/openshift3/ose-deployer        v3.10.0-0.29.0      ca787a3c6f62        2 weeks ago         1.62 GB
registry.reg-aws.openshift.com:443/openshift3/ose-web-console     v3.10.0-0.29.0      47aa15dcc21f        2 weeks ago         470 MB
registry.reg-aws.openshift.com:443/openshift3/ose-control-plane   v3.10.0-0.29.0      2fde04e76b11        2 weeks ago         1.62 GB
registry.reg-aws.openshift.com:443/openshift3/ose-pod             v3.10.0-0.29.0      3cc11b1d60be        2 weeks ago         214 MB
registry.access.redhat.com/rhel7/etcd                             3.2.15    

However, I can not re-produce it on latest openshift-ansible-3.10.0-0.46.0.git.0.85c3afd.el7.noarch.rpm. So remove test blocker first. And I try to re-produce it on original version 0.41.0, but still can not re-produce because I can not let all images stay on 0.41.0.

qe-jliu-r39b-master-etcd-1.0516-lh9.qe.rhcloud.com | SUCCESS | rc=0 >>
REPOSITORY                                                                  TAG                 IMAGE ID            CREATED             SIZE
registry.reg-aws.openshift.com:443/openshift3/ose-node                      v3.10               f6bb2f11d92a        8 hours ago         1.2 GB
registry.reg-aws.openshift.com:443/openshift3/ose-deployer                  v3.10.0-0.46.0      12262ad818e0        8 hours ago         634 MB
registry.reg-aws.openshift.com:443/openshift3/ose-service-catalog           v3.10               4912c244e2ff        8 hours ago         310 MB
registry.reg-aws.openshift.com:443/openshift3/ose-web-console               v3.10               615eec2b9949        8 hours ago         318 MB
registry.reg-aws.openshift.com:443/openshift3/ose-control-plane             v3.10               35628f394eff        8 hours ago         634 MB
registry.reg-aws.openshift.com:443/openshift3/ose-template-service-broker   v3.10               e270f1d5200d        9 hours ago         284 MB
registry.reg-aws.openshift.com:443/openshift3/ose-ansible-service-broker    v3.10               8292e83fe982        9 hours ago         440 MB
registry.reg-aws.openshift.com:443/openshift3/registry-console              v3.10               eaca67e14beb        9 hours ago         243 MB
registry.reg-aws.openshift.com:443/openshift3/ose-pod                       v3.10.0-0.41.0      653bda7c8a49        4 days ago          214 MB
registry.reg-aws.openshift.com:443/openshift3/ose-deployer                  v3.9.29             17067913d670        6 days ago          1.23 GB
registry.reg-aws.openshift.com:443/openshift3/ose-web-console               v3.9.29             ae9986489fc7        6 days ago          466 MB
registry.reg-aws.openshift.com:443/openshift3/ose-pod                       v3.9.29             490f82693c58        7 days ago          214 MB
registry.reg-aws.openshift.com:443/openshift3/ose-template-service-broker   v3.9.29             e017c1d36a84        7 days ago          299 MB
registry.reg-aws.openshift.com:443/openshift3/ose-service-catalog           v3.9.29             6800c342ce7e        7 days ago          288 MB
registry.access.redhat.com/rhel7/etcd                                       latest              924412659272        7 days ago          256 MB
registry.access.redhat.com/rhel7/etcd                                       3.2.15              4f35b6516d22        5 weeks ago         256 MB
 
I will re-open it again if I hit it in later test, close it first.

Note You need to log in before you can comment on or make changes to this bug.