Description of problem: Template service broker still use ose:v3.7 image after upgrade from 3.7 to 3.9, but it should use ose-template-service-broker:v3.9: [root@host-172-16-120-29 ~]# oc get po -o yaml -n openshift-template-service-broker | grep -i image image: xxxxxxxxxxx:443/openshift3/ose:v3.7 imagePullPolicy: IfNotPresent imagePullSecrets: image: xxxxxxxxxxx:443/openshift3/ose:v3.7 imageID: docker-pullable://xxxxxxxxxxx:443/openshift3/ose@sha256:5ae218f6cb9989288d43c7c0d263b1dfa2ab192a3864d68e7c38514b1bc21b83 image: xxxxxxxxxxx:443/openshift3/ose:v3.7 imagePullPolicy: IfNotPresent imagePullSecrets: image: xxxxxxxxxxx:443/openshift3/ose:v3.7 imageID: docker-pullable://xxxxxxxxxxx:443/openshift3/ose@sha256:5ae218f6cb9989288d43c7c0d263b1dfa2ab192a3864d68e7c38514b1bc21b83 image: xxxxxxxxxxx:443/openshift3/ose:v3.7 imagePullPolicy: IfNotPresent imagePullSecrets: image: xxxxxxxxxxx:443/openshift3/ose:v3.7 imageID: docker-pullable://xxxxxxxxxxx:443/openshift3/ose@sha256:5ae218f6cb9989288d43c7c0d263b1dfa2ab192a3864d68e7c38514b1bc21b83 [root@host-172-16-120-29 ~]# openshift version openshift v3.9.0-0.34.0 kubernetes v1.9.1+a0ce1bc657 etcd 3.2.8 Version-Release number of selected component (if applicable): How reproducible: always Steps to Reproduce: 1.Upgrade openshift from v3.7 to v3.9 2.Check template service broker image 3. Actual results: TSB still use ose:v3.7 image Expected results: TSB should use openshift3/ose-template-service-broker:v3.9 after upgrade. Additional info: This template include image name and command when start TSB: https://github.com/openshift/origin/blob/master/install/templateservicebroker/apiserver-template.yaml
I assume we aren't doing anything to update the TSB on upgrade, will need to chat w/ Scott in the morning.
PR Merged: https://github.com/openshift/openshift-ansible/pull/7251
The verification is blocked by this bug now: https://bugzilla.redhat.com/show_bug.cgi?id=1547803
I looked at what happened in my 3.7 to 3.9 upgraded environment and it was in the same state. TSB 3.7 pod still running but the daemonset had been updated to use a 3.9 image. I then deleted the apiserver pod and it was re-created using 3.9 image. This is a shortcoming of daemonsets, there's no way to upgrade them. The options are to either toggle labels to unschedule and then re-schedule them or to simply delete the pod and allow it to be re-created. I discussed this with Jim Minter who originally did the TSB work and aside from a potential brief outage of TSB he indicates there's no harm in simply deleting the TSB pods. That's the best we can right now, we should look at migrating TSB to deployments which support rollout. [root@ose3-master ~]# oc status -n openshift-template-service-broker E0307 16:20:43.270947 59644 service_group.go:83] unrecognized container: DaemonSet|openshift-template-service-broker/apiserver (*nodes.DaemonSetNode) In project openshift-template-service-broker on server https://ose3-master.example.com:8443 svc/apiserver - 172.30.144.6:443 -> 8443 pod/apiserver-6zlc4 runs registry.access.redhat.com/openshift3/ose:v3.7 daemonset/apiserver manages registry.reg-aws.openshift.com/openshift3/ose-template-service-broker:v3.9.3 generation #2 running for 2 months - 1 pod 1 info identified, use 'oc status -v' to see details. [root@ose3-master ~]# oc delete po/apiserver-6zlc4 pod "apiserver-6zlc4" deleted [root@ose3-master ~]# oc get pods NAME READY STATUS RESTARTS AGE apiserver-6zlc4 0/1 Terminating 3 77d [root@ose3-master ~]# oc status E0307 16:22:57.581256 60865 service_group.go:83] unrecognized container: DaemonSet|openshift-template-service-broker/apiserver (*nodes.DaemonSetNode) In project openshift-template-service-broker on server https://ose3-master.example.com:8443 svc/apiserver - 172.30.144.6:443 -> 8443 pod/apiserver-6zlc4 runs registry.access.redhat.com/openshift3/ose:v3.7 daemonset/apiserver manages registry.reg-aws.openshift.com/openshift3/ose-template-service-broker:v3.9.3 generation #2 running for 2 months - 0/1 pods growing to 1 1 info identified, use 'oc status -v' to see details. [root@ose3-master ~]# oc status E0307 16:23:03.518046 61023 service_group.go:83] unrecognized container: DaemonSet|openshift-template-service-broker/apiserver (*nodes.DaemonSetNode) In project openshift-template-service-broker on server https://ose3-master.example.com:8443 svc/apiserver - 172.30.144.6:443 -> 8443 pod/apiserver-qrnff runs registry.reg-aws.openshift.com/openshift3/ose-template-service-broker:v3.9.3 daemonset/apiserver manages registry.reg-aws.openshift.com/openshift3/ose-template-service-broker:v3.9.3 generation #2 running for 2 months - 0/1 pods growing to 1
PR Created: https://github.com/openshift/openshift-ansible/pull/7433
Verification of this bug is blocked by the new bug: https://bugzilla.redhat.com/show_bug.cgi?id=1553624
Sorry, I found upgrade install and fresh install are using difference image prefix, the bug should be fixed.
Please ignore comment #11. Per comment #10, moving to verified. $oc get pods -n openshift-template-service-broker -o yaml | grep image image: brew-pulp-xxxxxxxxxxxx:8888/openshift3/template-service-broker:v3.9.7
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:0489