Bug 1658387
Summary: | Updating from 3.11.16 to 3.11.43 does not pull in 3.11.43 images unless you specify openshift_image_tag=v3.11.43 in inventory | |||
---|---|---|---|---|
Product: | OpenShift Container Platform | Reporter: | Candace Sheremeta <cshereme> | |
Component: | Installer | Assignee: | Patrick Dillon <padillon> | |
Installer sub component: | openshift-ansible | QA Contact: | liujia <jiajliu> | |
Status: | CLOSED ERRATA | Docs Contact: | ||
Severity: | high | |||
Priority: | high | CC: | aos-bugs, dmoessne, gpei, jiajliu, jokerman, jrosenta, mmccomas, palonsor, rsandu, wmeng, wzheng | |
Version: | 3.11.0 | |||
Target Milestone: | --- | |||
Target Release: | 3.11.z | |||
Hardware: | Unspecified | |||
OS: | Unspecified | |||
Whiteboard: | ||||
Fixed In Version: | Doc Type: | Bug Fix | ||
Doc Text: |
Cause: playbooks ran a check to see if image existed on disk with X-version tag, but did not make sure the version on disk was up-to-date to the tagged image in the repo. If the image existed on disk, the image pull was skipped.
Consequence: Z-version updates would not work. If trying to upgrade from, e.g. 3.11.16 to 3.11.43 the 3.11.16 image would satisfy the check and the 3.11.43 image would not be pulled. Therefore the cluster would remain at 3.11.16 before and after the upgrade.
Fix: Remove the on disk check. Image pulls are efficient so there is no need to check whether the disk exists on disk before hand.
Result: Upgrades proceed as expected.
|
Story Points: | --- | |
Clone Of: | ||||
: | 1659600 (view as bug list) | Environment: | ||
Last Closed: | 2019-04-11 05:38:23 UTC | Type: | Bug | |
Regression: | --- | Mount Type: | --- | |
Documentation: | --- | CRM: | ||
Verified Versions: | Category: | --- | ||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
Cloudforms Team: | --- | Target Upstream Version: | ||
Embargoed: | ||||
Bug Depends On: | ||||
Bug Blocks: | 1659600, 1684142 |
Description
Candace Sheremeta
2018-12-11 22:54:20 UTC
Workaround is to set openshift_image_tag to the desired image tag to be used by control plane components. ie: openshift_image_tag=v3.11.43 *** Bug 1657737 has been marked as a duplicate of this bug. *** This is also being seen with 3.11.51 updates - the new images are not being pulled down *** Bug 1662036 has been marked as a duplicate of this bug. *** The conditional check before pulling images would allow stale images to remain instead of pulling the most recent image for an upgrade. I removed the condition. PR in 3.11 branch: https://github.com/openshift/openshift-ansible/pull/11108 This PR fixes a problem in previous PR, where if etcd pre-pull was skipped (e.g. standalone etcd cluster) then the async status check would still run and fail on undefined variable etcd_prepull: https://github.com/openshift/openshift-ansible/pull/11151 Reproduce on openshift-ansible-3.11.72-1.git.0.7c8b4f0.el7.noarch Before upgrade: [root@ip-172-18-2-44 ~]# openshift version openshift v3.11.69 [root@ip-172-18-2-44 ~]# docker images|grep ose registry.reg-aws.openshift.com:443/openshift3/ose-console v3.11 12c04fa153e5 13 days ago 254 MB registry.reg-aws.openshift.com:443/openshift3/ose-node v3.11 ffdbad0af1ff 2 weeks ago 1.17 GB registry.reg-aws.openshift.com:443/openshift3/ose-deployer v3.11.69 1a8c15715a72 2 weeks ago 361 MB registry.reg-aws.openshift.com:443/openshift3/ose-kube-rbac-proxy v3.11 e8b055111fb8 2 weeks ago 418 MB registry.reg-aws.openshift.com:443/openshift3/ose-service-catalog v3.11 56644d30e67e 2 weeks ago 309 MB registry.reg-aws.openshift.com:443/openshift3/ose-template-service-broker v3.11 eed48c932cfa 2 weeks ago 313 MB registry.reg-aws.openshift.com:443/openshift3/ose-web-console v3.11 ebd8d8c2144b 2 weeks ago 322 MB registry.reg-aws.openshift.com:443/openshift3/ose-pod v3.11.69 298223cef55e 2 weeks ago 238 MB registry.reg-aws.openshift.com:443/openshift3/ose-control-plane v3.11 fd79ae3b8725 3 weeks ago 808 MB [root@ip-172-18-2-44 ~]# After upgrade: [root@ip-172-18-2-44 ~]# openshift version openshift v3.11.72 [root@ip-172-18-2-44 ~]# docker images|grep ose registry.reg-aws.openshift.com:443/openshift3/ose-pod v3.11 74ee8ca2c756 9 hours ago 238 MB registry.reg-aws.openshift.com:443/openshift3/ose-console v3.11 12c04fa153e5 13 days ago 254 MB registry.reg-aws.openshift.com:443/openshift3/ose-console v3.11.69 12c04fa153e5 13 days ago 254 MB registry.reg-aws.openshift.com:443/openshift3/ose-node v3.11 ffdbad0af1ff 2 weeks ago 1.17 GB registry.reg-aws.openshift.com:443/openshift3/ose-node v3.11.69 ffdbad0af1ff 2 weeks ago 1.17 GB registry.reg-aws.openshift.com:443/openshift3/ose-deployer v3.11.69 1a8c15715a72 2 weeks ago 361 MB registry.reg-aws.openshift.com:443/openshift3/ose-kube-rbac-proxy v3.11 e8b055111fb8 2 weeks ago 418 MB registry.reg-aws.openshift.com:443/openshift3/ose-kube-rbac-proxy v3.11.69 e8b055111fb8 2 weeks ago 418 MB registry.reg-aws.openshift.com:443/openshift3/ose-service-catalog v3.11 56644d30e67e 2 weeks ago 309 MB registry.reg-aws.openshift.com:443/openshift3/ose-service-catalog v3.11.69 56644d30e67e 2 weeks ago 309 MB registry.reg-aws.openshift.com:443/openshift3/ose-template-service-broker v3.11 eed48c932cfa 2 weeks ago 313 MB registry.reg-aws.openshift.com:443/openshift3/ose-template-service-broker v3.11.69 eed48c932cfa 2 weeks ago 313 MB registry.reg-aws.openshift.com:443/openshift3/ose-web-console v3.11 ebd8d8c2144b 2 weeks ago 322 MB registry.reg-aws.openshift.com:443/openshift3/ose-web-console v3.11.69 ebd8d8c2144b 2 weeks ago 322 MB registry.reg-aws.openshift.com:443/openshift3/ose-pod v3.11.69 298223cef55e 2 weeks ago 238 MB registry.reg-aws.openshift.com:443/openshift3/ose-pod v3.11.72 941e32701d4b 3 weeks ago 238 MB registry.reg-aws.openshift.com:443/openshift3/ose-control-plane v3.11 fd79ae3b8725 3 weeks ago 808 MB Verified on openshift-ansible-3.11.82-1.git.0.f29227a.el7.noarch Before upgrade: [root@ip-172-18-10-66 ~]# openshift version openshift v3.11.69 [root@ip-172-18-10-66 ~]# docker images|grep ose registry.reg-aws.openshift.com:443/openshift3/ose-console v3.11 12c04fa153e5 13 days ago 254 MB registry.reg-aws.openshift.com:443/openshift3/ose-node v3.11 ffdbad0af1ff 2 weeks ago 1.17 GB registry.reg-aws.openshift.com:443/openshift3/ose-deployer v3.11.69 1a8c15715a72 2 weeks ago 361 MB registry.reg-aws.openshift.com:443/openshift3/ose-kube-rbac-proxy v3.11 e8b055111fb8 2 weeks ago 418 MB registry.reg-aws.openshift.com:443/openshift3/ose-service-catalog v3.11 56644d30e67e 2 weeks ago 309 MB registry.reg-aws.openshift.com:443/openshift3/ose-template-service-broker v3.11 eed48c932cfa 2 weeks ago 313 MB registry.reg-aws.openshift.com:443/openshift3/ose-web-console v3.11 ebd8d8c2144b 2 weeks ago 322 MB registry.reg-aws.openshift.com:443/openshift3/ose-pod v3.11.69 298223cef55e 2 weeks ago 238 MB registry.reg-aws.openshift.com:443/openshift3/ose-control-plane v3.11 fd79ae3b8725 3 weeks ago 808 MB After upgrade: [root@ip-172-18-10-66 ~]# openshift version openshift v3.11.82 [root@ip-172-18-10-66 ~]# docker images|grep ose registry.reg-aws.openshift.com:443/openshift3/ose-node v3.11 6fc64a6096d1 8 hours ago 1.17 GB registry.reg-aws.openshift.com:443/openshift3/ose-control-plane v3.11 af096add3983 9 hours ago 808 MB registry.reg-aws.openshift.com:443/openshift3/ose-deployer v3.11.82 54d335969fb7 9 hours ago 361 MB registry.reg-aws.openshift.com:443/openshift3/ose-pod v3.11 74ee8ca2c756 9 hours ago 238 MB registry.reg-aws.openshift.com:443/openshift3/ose-pod v3.11.82 74ee8ca2c756 9 hours ago 238 MB registry.reg-aws.openshift.com:443/openshift3/ose-console v3.11 12c04fa153e5 13 days ago 254 MB registry.reg-aws.openshift.com:443/openshift3/ose-console v3.11.69 12c04fa153e5 13 days ago 254 MB registry.reg-aws.openshift.com:443/openshift3/ose-node <none> ffdbad0af1ff 2 weeks ago 1.17 GB registry.reg-aws.openshift.com:443/openshift3/ose-deployer v3.11.69 1a8c15715a72 2 weeks ago 361 MB registry.reg-aws.openshift.com:443/openshift3/ose-kube-rbac-proxy v3.11 e8b055111fb8 2 weeks ago 418 MB registry.reg-aws.openshift.com:443/openshift3/ose-kube-rbac-proxy v3.11.69 e8b055111fb8 2 weeks ago 418 MB registry.reg-aws.openshift.com:443/openshift3/ose-service-catalog v3.11 56644d30e67e 2 weeks ago 309 MB registry.reg-aws.openshift.com:443/openshift3/ose-service-catalog v3.11.69 56644d30e67e 2 weeks ago 309 MB registry.reg-aws.openshift.com:443/openshift3/ose-template-service-broker v3.11 eed48c932cfa 2 weeks ago 313 MB registry.reg-aws.openshift.com:443/openshift3/ose-template-service-broker v3.11.69 eed48c932cfa 2 weeks ago 313 MB registry.reg-aws.openshift.com:443/openshift3/ose-web-console v3.11 ebd8d8c2144b 2 weeks ago 322 MB registry.reg-aws.openshift.com:443/openshift3/ose-web-console v3.11.69 ebd8d8c2144b 2 weeks ago 322 MB registry.reg-aws.openshift.com:443/openshift3/ose-pod v3.11.69 298223cef55e 2 weeks ago 238 MB registry.reg-aws.openshift.com:443/openshift3/ose-control-plane <none> fd79ae3b8725 3 weeks ago 808 MB ose-node. ose-control-plane, ose-deployer, ose-pod image updated to latest. but still some images keep old v3.11.69 version, such as ose-console, ose-web-console, ose-template-service-broker, ose-service-catalog, ose-kube-rbac-proxy Checked that the latest image is available, take ose-web-console for example: [root@ip-172-18-10-66 ~]# docker pull registry.reg-aws.openshift.com:443/openshift3/ose-web-console:v3.11 Trying to pull repository registry.reg-aws.openshift.com:443/openshift3/ose-web-console ... v3.11: Pulling from registry.reg-aws.openshift.com:443/openshift3/ose-web-console c325120ebc8d: Already exists c9d123037991: Already exists 923f97b65649: Already exists 0d1c719b5461: Pull complete Digest: sha256:c0ef4c9dbbea0d192a8cc9b2350ac71bd43f2dc1887f1179696cd294454b2058 Status: Downloaded newer image for registry.reg-aws.openshift.com:443/openshift3/ose-web-console:v3.11 [root@ip-172-18-10-66 ~]# docker images|grep ose-web registry.reg-aws.openshift.com:443/openshift3/ose-web-console v3.11 4a5379481b8c 10 hours ago 322 MB registry.reg-aws.openshift.com:443/openshift3/ose-web-console v3.11.69 ebd8d8c2144b 2 weeks ago 322 MB According to above, assign the bug back We'd like to split this into two bugs, one that addresses the control plane static pods and one that addresses items deployed on top of the cluster ie: web-console, etc. I'll create a new bug for the second portion of the problem and we'll move this one to ON_QA to verify the control plane static pods. new bug: https://bugzilla.redhat.com/show_bug.cgi?id=1684142 for 3.10: https://bugzilla.redhat.com/show_bug.cgi?id=1684315 According to comment 15 and comment 21, agree to split the bug to two separate issues. So for this bug, the control plane static pods were updated during minor upgrade. Verify the bug, will keep left parts tracking in bug1684142. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0636 |