Bug 1658387 - Updating from 3.11.16 to 3.11.43 does not pull in 3.11.43 images unless you specify openshift_image_tag=v3.11.43 in inventory
Summary: Updating from 3.11.16 to 3.11.43 does not pull in 3.11.43 images unless you s...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer
Version: 3.11.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 3.11.z
Assignee: Patrick Dillon
QA Contact: liujia
URL:
Whiteboard:
: 1657737 1662036 (view as bug list)
Depends On:
Blocks: 1659600 1684142
TreeView+ depends on / blocked
 
Reported: 2018-12-11 22:54 UTC by Candace Sheremeta
Modified: 2022-03-13 16:27 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: playbooks ran a check to see if image existed on disk with X-version tag, but did not make sure the version on disk was up-to-date to the tagged image in the repo. If the image existed on disk, the image pull was skipped. Consequence: Z-version updates would not work. If trying to upgrade from, e.g. 3.11.16 to 3.11.43 the 3.11.16 image would satisfy the check and the 3.11.43 image would not be pulled. Therefore the cluster would remain at 3.11.16 before and after the upgrade. Fix: Remove the on disk check. Image pulls are efficient so there is no need to check whether the disk exists on disk before hand. Result: Upgrades proceed as expected.
Clone Of:
: 1659600 (view as bug list)
Environment:
Last Closed: 2019-04-11 05:38:23 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 3774891 0 None None None 2018-12-25 18:31:21 UTC
Red Hat Product Errata RHBA-2019:0636 0 None None None 2019-04-11 05:38:38 UTC

Description Candace Sheremeta 2018-12-11 22:54:20 UTC 
Description of problem:
Updating from 3.11.16 to 3.11.43 does not pull in 3.11.43 images unless you specify openshift_image_tag=v.3.11.43 in inventory. After updating openshift-ansible to 3.11.43 and running the upgrade playbook against a 3.11.16 cluster to upgrade to 3.11.43, all pods still use 3.11.16 image versions.

More info from customer to be provided in private comment

Version-Release number of selected component (if applicable):
3.11.43


Expected results:
Upgrade to 3.11.43 pulls down 3.11.43 images
Comment 2 Scott Dodson 2018-12-12 13:39:39 UTC 
Workaround is to set openshift_image_tag to the desired image tag to be used by control plane components. ie:

openshift_image_tag=v3.11.43
Comment 4 Scott Dodson 2018-12-13 14:53:50 UTC 
*** Bug 1657737 has been marked as a duplicate of this bug. ***
Comment 5 Candace Sheremeta 2018-12-14 01:32:09 UTC 
This is also being seen with 3.11.51 updates - the new images are not being pulled down
Comment 8 Scott Dodson 2019-01-02 13:34:10 UTC 
*** Bug 1662036 has been marked as a duplicate of this bug. ***
Comment 11 Patrick Dillon 2019-01-31 21:35:55 UTC 
The conditional check before pulling images would allow stale images to remain instead of pulling the most recent image for an upgrade. I removed the condition.

PR in 3.11 branch: https://github.com/openshift/openshift-ansible/pull/11108
Comment 12 Patrick Dillon 2019-02-08 14:41:00 UTC 
This PR fixes a problem in previous PR, where if etcd pre-pull was skipped (e.g. standalone etcd cluster) then the async status check would still run and fail on undefined variable etcd_prepull:
https://github.com/openshift/openshift-ansible/pull/11151
Comment 14 liujia 2019-02-13 02:15:04 UTC 
Reproduce on openshift-ansible-3.11.72-1.git.0.7c8b4f0.el7.noarch

Before upgrade:
[root@ip-172-18-2-44 ~]# openshift version
openshift v3.11.69
[root@ip-172-18-2-44 ~]# docker images|grep ose
registry.reg-aws.openshift.com:443/openshift3/ose-console                   v3.11               12c04fa153e5        13 days ago         254 MB
registry.reg-aws.openshift.com:443/openshift3/ose-node                      v3.11               ffdbad0af1ff        2 weeks ago         1.17 GB
registry.reg-aws.openshift.com:443/openshift3/ose-deployer                  v3.11.69            1a8c15715a72        2 weeks ago         361 MB
registry.reg-aws.openshift.com:443/openshift3/ose-kube-rbac-proxy           v3.11               e8b055111fb8        2 weeks ago         418 MB
registry.reg-aws.openshift.com:443/openshift3/ose-service-catalog           v3.11               56644d30e67e        2 weeks ago         309 MB
registry.reg-aws.openshift.com:443/openshift3/ose-template-service-broker   v3.11               eed48c932cfa        2 weeks ago         313 MB
registry.reg-aws.openshift.com:443/openshift3/ose-web-console               v3.11               ebd8d8c2144b        2 weeks ago         322 MB
registry.reg-aws.openshift.com:443/openshift3/ose-pod                       v3.11.69            298223cef55e        2 weeks ago         238 MB
registry.reg-aws.openshift.com:443/openshift3/ose-control-plane             v3.11               fd79ae3b8725        3 weeks ago         808 MB
[root@ip-172-18-2-44 ~]# 

After upgrade:
[root@ip-172-18-2-44 ~]# openshift version
openshift v3.11.72
[root@ip-172-18-2-44 ~]# docker images|grep ose
registry.reg-aws.openshift.com:443/openshift3/ose-pod                       v3.11               74ee8ca2c756        9 hours ago         238 MB
registry.reg-aws.openshift.com:443/openshift3/ose-console                   v3.11               12c04fa153e5        13 days ago         254 MB
registry.reg-aws.openshift.com:443/openshift3/ose-console                   v3.11.69            12c04fa153e5        13 days ago         254 MB
registry.reg-aws.openshift.com:443/openshift3/ose-node                      v3.11               ffdbad0af1ff        2 weeks ago         1.17 GB
registry.reg-aws.openshift.com:443/openshift3/ose-node                      v3.11.69            ffdbad0af1ff        2 weeks ago         1.17 GB
registry.reg-aws.openshift.com:443/openshift3/ose-deployer                  v3.11.69            1a8c15715a72        2 weeks ago         361 MB
registry.reg-aws.openshift.com:443/openshift3/ose-kube-rbac-proxy           v3.11               e8b055111fb8        2 weeks ago         418 MB
registry.reg-aws.openshift.com:443/openshift3/ose-kube-rbac-proxy           v3.11.69            e8b055111fb8        2 weeks ago         418 MB
registry.reg-aws.openshift.com:443/openshift3/ose-service-catalog           v3.11               56644d30e67e        2 weeks ago         309 MB
registry.reg-aws.openshift.com:443/openshift3/ose-service-catalog           v3.11.69            56644d30e67e        2 weeks ago         309 MB
registry.reg-aws.openshift.com:443/openshift3/ose-template-service-broker   v3.11               eed48c932cfa        2 weeks ago         313 MB
registry.reg-aws.openshift.com:443/openshift3/ose-template-service-broker   v3.11.69            eed48c932cfa        2 weeks ago         313 MB
registry.reg-aws.openshift.com:443/openshift3/ose-web-console               v3.11               ebd8d8c2144b        2 weeks ago         322 MB
registry.reg-aws.openshift.com:443/openshift3/ose-web-console               v3.11.69            ebd8d8c2144b        2 weeks ago         322 MB
registry.reg-aws.openshift.com:443/openshift3/ose-pod                       v3.11.69            298223cef55e        2 weeks ago         238 MB
registry.reg-aws.openshift.com:443/openshift3/ose-pod                       v3.11.72            941e32701d4b        3 weeks ago         238 MB
registry.reg-aws.openshift.com:443/openshift3/ose-control-plane             v3.11               fd79ae3b8725        3 weeks ago         808 MB
Comment 15 liujia 2019-02-13 03:42:47 UTC 
Verified on openshift-ansible-3.11.82-1.git.0.f29227a.el7.noarch

Before upgrade:
[root@ip-172-18-10-66 ~]# openshift version
openshift v3.11.69
[root@ip-172-18-10-66 ~]# docker images|grep ose
registry.reg-aws.openshift.com:443/openshift3/ose-console                   v3.11               12c04fa153e5        13 days ago         254 MB
registry.reg-aws.openshift.com:443/openshift3/ose-node                      v3.11               ffdbad0af1ff        2 weeks ago         1.17 GB
registry.reg-aws.openshift.com:443/openshift3/ose-deployer                  v3.11.69            1a8c15715a72        2 weeks ago         361 MB
registry.reg-aws.openshift.com:443/openshift3/ose-kube-rbac-proxy           v3.11               e8b055111fb8        2 weeks ago         418 MB
registry.reg-aws.openshift.com:443/openshift3/ose-service-catalog           v3.11               56644d30e67e        2 weeks ago         309 MB
registry.reg-aws.openshift.com:443/openshift3/ose-template-service-broker   v3.11               eed48c932cfa        2 weeks ago         313 MB
registry.reg-aws.openshift.com:443/openshift3/ose-web-console               v3.11               ebd8d8c2144b        2 weeks ago         322 MB
registry.reg-aws.openshift.com:443/openshift3/ose-pod                       v3.11.69            298223cef55e        2 weeks ago         238 MB
registry.reg-aws.openshift.com:443/openshift3/ose-control-plane             v3.11               fd79ae3b8725        3 weeks ago         808 MB

After upgrade:
[root@ip-172-18-10-66 ~]# openshift version
openshift v3.11.82
[root@ip-172-18-10-66 ~]# docker images|grep ose
registry.reg-aws.openshift.com:443/openshift3/ose-node                      v3.11               6fc64a6096d1        8 hours ago         1.17 GB
registry.reg-aws.openshift.com:443/openshift3/ose-control-plane             v3.11               af096add3983        9 hours ago         808 MB
registry.reg-aws.openshift.com:443/openshift3/ose-deployer                  v3.11.82            54d335969fb7        9 hours ago         361 MB
registry.reg-aws.openshift.com:443/openshift3/ose-pod                       v3.11               74ee8ca2c756        9 hours ago         238 MB
registry.reg-aws.openshift.com:443/openshift3/ose-pod                       v3.11.82            74ee8ca2c756        9 hours ago         238 MB
registry.reg-aws.openshift.com:443/openshift3/ose-console                   v3.11               12c04fa153e5        13 days ago         254 MB
registry.reg-aws.openshift.com:443/openshift3/ose-console                   v3.11.69            12c04fa153e5        13 days ago         254 MB
registry.reg-aws.openshift.com:443/openshift3/ose-node                      <none>              ffdbad0af1ff        2 weeks ago         1.17 GB
registry.reg-aws.openshift.com:443/openshift3/ose-deployer                  v3.11.69            1a8c15715a72        2 weeks ago         361 MB
registry.reg-aws.openshift.com:443/openshift3/ose-kube-rbac-proxy           v3.11               e8b055111fb8        2 weeks ago         418 MB
registry.reg-aws.openshift.com:443/openshift3/ose-kube-rbac-proxy           v3.11.69            e8b055111fb8        2 weeks ago         418 MB
registry.reg-aws.openshift.com:443/openshift3/ose-service-catalog           v3.11               56644d30e67e        2 weeks ago         309 MB
registry.reg-aws.openshift.com:443/openshift3/ose-service-catalog           v3.11.69            56644d30e67e        2 weeks ago         309 MB
registry.reg-aws.openshift.com:443/openshift3/ose-template-service-broker   v3.11               eed48c932cfa        2 weeks ago         313 MB
registry.reg-aws.openshift.com:443/openshift3/ose-template-service-broker   v3.11.69            eed48c932cfa        2 weeks ago         313 MB
registry.reg-aws.openshift.com:443/openshift3/ose-web-console               v3.11               ebd8d8c2144b        2 weeks ago         322 MB
registry.reg-aws.openshift.com:443/openshift3/ose-web-console               v3.11.69            ebd8d8c2144b        2 weeks ago         322 MB
registry.reg-aws.openshift.com:443/openshift3/ose-pod                       v3.11.69            298223cef55e        2 weeks ago         238 MB
registry.reg-aws.openshift.com:443/openshift3/ose-control-plane             <none>              fd79ae3b8725        3 weeks ago         808 MB


ose-node. ose-control-plane, ose-deployer, ose-pod image updated to latest.
but still some images keep old v3.11.69 version, such as ose-console, ose-web-console, ose-template-service-broker, ose-service-catalog, ose-kube-rbac-proxy

Checked that the latest image is available, take ose-web-console for example:
[root@ip-172-18-10-66 ~]# docker pull registry.reg-aws.openshift.com:443/openshift3/ose-web-console:v3.11
Trying to pull repository registry.reg-aws.openshift.com:443/openshift3/ose-web-console ... 
v3.11: Pulling from registry.reg-aws.openshift.com:443/openshift3/ose-web-console
c325120ebc8d: Already exists 
c9d123037991: Already exists 
923f97b65649: Already exists 
0d1c719b5461: Pull complete 
Digest: sha256:c0ef4c9dbbea0d192a8cc9b2350ac71bd43f2dc1887f1179696cd294454b2058
Status: Downloaded newer image for registry.reg-aws.openshift.com:443/openshift3/ose-web-console:v3.11

[root@ip-172-18-10-66 ~]# docker images|grep ose-web
registry.reg-aws.openshift.com:443/openshift3/ose-web-console               v3.11               4a5379481b8c        10 hours ago        322 MB
registry.reg-aws.openshift.com:443/openshift3/ose-web-console               v3.11.69            ebd8d8c2144b        2 weeks ago         322 MB

According to above, assign the bug back
Comment 21 Scott Dodson 2019-02-28 14:37:35 UTC 
We'd like to split this into two bugs, one that addresses the control plane static pods and one that addresses items deployed on top of the cluster ie: web-console, etc. I'll create a new bug for the second portion of the problem and we'll move this one to ON_QA to verify the control plane static pods.
Comment 23 Weihua Meng 2019-03-01 01:01:41 UTC 
new bug: https://bugzilla.redhat.com/show_bug.cgi?id=1684142 

for 3.10: https://bugzilla.redhat.com/show_bug.cgi?id=1684315
Comment 24 liujia 2019-03-01 02:10:13 UTC 
According to comment 15 and comment 21, agree to split the bug to two separate issues. So for this bug, the control plane static pods were updated during minor upgrade. Verify the bug, will keep left parts tracking in bug1684142.
Comment 27 errata-xmlrpc 2019-04-11 05:38:23 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0636
Note You need to log in before you can comment on or make changes to this bug.