Description of problem: Failed to upgrade from 3.7 to 3.9 for project openshift-template-service-broker is deleted: Before upgrade: [root@host-172-16-120-56 ~]# openshift version openshift v3.7.36 kubernetes v1.7.6+a08f5eeb62 etcd 3.2.8 [root@host-172-16-120-56 ~]# oc get pods -n openshift-template-service-broker NAME READY STATUS RESTARTS AGE apiserver-dlmnm 1/1 Running 0 54s apiserver-x9k5c 1/1 Running 0 54s After upgrade: [root@host-172-16-120-56 ~]# openshift version openshift v3.9.4 kubernetes v1.9.1+a0ce1bc657 etcd 3.2.16 [root@host-172-16-120-56 ~]# oc get projects NAME DISPLAY NAME STATUS default Active install-test Active kube-public Active kube-service-catalog Active kube-system Active logging Active management-infra Active openshift Active openshift-ansible-service-broker Active openshift-infra Active openshift-node Active openshift-web-console Active Here is detailed log: fatal: [xxxxxxxxxxx]: FAILED! => {"changed": true, "cmd": "oc process --config=/tmp/tsb-ansible-ZFYKr4/admin.kubeconfig -f \"/tmp/tsb-ansible-ZFYKr4/apiserver-template.yaml\" --param API_SERVER_CONFIG=\"kind: TemplateServiceBrokerConfig\napiVersion: config.templateservicebroker.openshift.io/v1\ntemplateNamespaces:\n- openshift\n\" --param IMAGE=\"registry.reg-aws.openshift.com:443/openshift3/template-service-broker:v3.9.4\" --param NODE_SELECTOR='{\"role\": \"node\"}' | oc apply --config=/tmp/tsb-ansible-ZFYKr4/admin.kubeconfig -f -", "delta": "0:00:00.755548", "end": "2018-03-09 02:57:52.015555", "msg": "non-zero return code", "rc": 1, "start": "2018-03-09 02:57:51.260007", "stderr": "Error from server (Forbidden): daemonsets.extensions \"apiserver\" is forbidden: unable to create new content in namespace openshift-template-service-broker because it is being terminated.\nError from server (Forbidden): configmaps \"apiserver-config\" is forbidden: unable to create new content in namespace openshift-template-service-broker because it is being terminated.\nError from server (Forbidden): serviceaccounts \"apiserver\" is forbidden: unable to create new content in namespace openshift-template-service-broker because it is being terminated.\nError from server (Forbidden): services \"apiserver\" is forbidden: unable to create new content in namespace openshift-template-service-broker because it is being terminated.\nError from server (Forbidden): serviceaccounts \"templateservicebroker-client\" is forbidden: unable to create new content in namespace openshift-template-service-broker because it is being terminated.\nError from server (Forbidden): secrets \"templateservicebroker-client\" is forbidden: unable to create new content in namespace openshift-template-service-broker because it is being terminated.", "stderr_lines": ["Error from server (Forbidden): daemonsets.extensions \"apiserver\" is forbidden: unable to create new content in namespace openshift-template-service-b roker because it is being terminated.", "Error from server (Forbidden): configmaps \"apiserver-config\" is forbidden: unable to create new content in namespace openshift-template-service-broker because it is being terminated.", "Error from server (Forbidden): serviceaccounts \"apiserver\" is forbidden: unable to create new content in namespace openshift-template-service-broker because it is being terminated.", "Error from server (Forbidden): services \"apiserver\" is forbidden: unable to create new content in namespace openshift-template-service-broker because it is being terminated.", "Error from server (Forbidden): serviceaccounts \"templateservicebroker-client\" is forbidden: unable to create new content in namespace openshift-template-service-broker because it is being terminated.", "Error from server (Forbidden): secrets \"templateservicebroker-client\" is forbidden: unable to create new content in namespace openshift-template-service-broker because it is being terminated."], "stdout": "", "stdout_lines": []} Version-Release number of selected component (if applicable): openshift-ansible: 3.9.4-1 How reproducible: always Steps to Reproduce: 1.Set up a 3.7 env with service catalog enabled 2.Upgrade to 3.9 3. Actual results: Failed to upgrade for unable to create new content in namespace openshift-template-service-broker because it is being terminated. Expected results: should upgrade to 3.9 Additional info: This should be introduced by this pr: https://github.com/openshift/openshift-ansible/pull/7433 which tries to fix bug https://bugzilla.redhat.com/show_bug.cgi?id=1540521
This may due to during project openshift-template-service-broker being deleting, new deployment happens in the same time. So if there is a validation to check whether the project has been deleted before new deploy here: https://github.com/openshift/openshift-ansible/blob/master/roles/template_service_broker/tasks/upgrade.yml, should no this issue.
Reverting the upgrade deletion step. PR Created: https://github.com/openshift/openshift-ansible/pull/7474
release-3.9 backport https://github.com/openshift/openshift-ansible/pull/7480
Verified with openshift-ansible:3.9.7-1, no such error when upgrade now.