Bug 1553624 - Failed to upgrade from 3.7 to 3.9 for cannot find project openshift-template-service-broker
Summary: Failed to upgrade from 3.7 to 3.9 for cannot find project openshift-template-...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Cluster Version Operator
Version: 3.9.0
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 3.9.0
Assignee: Michael Gugino
QA Contact: Wenjing Zheng
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-03-09 08:18 UTC by Wenjing Zheng
Modified: 2018-06-18 18:29 UTC (History)
7 users (show)

Fixed In Version: openshift-ansible:3.9.7-1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-06-18 18:07:06 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Wenjing Zheng 2018-03-09 08:18:57 UTC
Description of problem:
Failed to upgrade from 3.7 to 3.9 for project openshift-template-service-broker is deleted:
Before upgrade:
[root@host-172-16-120-56 ~]# openshift version
openshift v3.7.36
kubernetes v1.7.6+a08f5eeb62
etcd 3.2.8
[root@host-172-16-120-56 ~]# oc get pods -n openshift-template-service-broker
NAME              READY     STATUS    RESTARTS   AGE
apiserver-dlmnm   1/1       Running   0          54s
apiserver-x9k5c   1/1       Running   0          54s

After upgrade:
[root@host-172-16-120-56 ~]# openshift version
openshift v3.9.4
kubernetes v1.9.1+a0ce1bc657
etcd 3.2.16
[root@host-172-16-120-56 ~]# oc get projects
NAME                               DISPLAY NAME   STATUS
default                                           Active
install-test                                      Active
kube-public                                       Active
kube-service-catalog                              Active
kube-system                                       Active
logging                                           Active
management-infra                                  Active
openshift                                         Active
openshift-ansible-service-broker                  Active
openshift-infra                                   Active
openshift-node                                    Active
openshift-web-console                             Active

Here is detailed log:
fatal: [xxxxxxxxxxx]: FAILED! => {"changed": true, "cmd": "oc process --config=/tmp/tsb-ansible-ZFYKr4/admin.kubeconfig -f \"/tmp/tsb-ansible-ZFYKr4/apiserver-template.yaml\" --param API_SERVER_CONFIG=\"kind: TemplateServiceBrokerConfig\napiVersion: config.templateservicebroker.openshift.io/v1\ntemplateNamespaces:\n- openshift\n\" --param IMAGE=\"registry.reg-aws.openshift.com:443/openshift3/template-service-broker:v3.9.4\" --param NODE_SELECTOR='{\"role\": \"node\"}' | oc apply --config=/tmp/tsb-ansible-ZFYKr4/admin.kubeconfig -f -", "delta": "0:00:00.755548", "end": "2018-03-09 02:57:52.015555", "msg": "non-zero return code", "rc": 1, "start": "2018-03-09 02:57:51.260007", "stderr": "Error from server (Forbidden): daemonsets.extensions \"apiserver\" is forbidden: unable to create new content in namespace openshift-template-service-broker because it is being terminated.\nError from server (Forbidden): configmaps \"apiserver-config\" is forbidden: unable to create new content in namespace openshift-template-service-broker because it is being terminated.\nError from server (Forbidden): serviceaccounts \"apiserver\" is forbidden: unable to create new content in namespace openshift-template-service-broker because it is being terminated.\nError from server (Forbidden): services \"apiserver\" is forbidden: unable to create new content in namespace openshift-template-service-broker because it is being terminated.\nError from server (Forbidden): serviceaccounts \"templateservicebroker-client\" is forbidden: unable to create new content in namespace openshift-template-service-broker because it is being terminated.\nError from server (Forbidden): secrets \"templateservicebroker-client\" is forbidden: unable to create new content in namespace openshift-template-service-broker because it is being terminated.", "stderr_lines": ["Error from server (Forbidden): daemonsets.extensions \"apiserver\" is forbidden: unable to create new content in namespace openshift-template-service-b
roker because it is being terminated.", "Error from server (Forbidden): configmaps \"apiserver-config\" is forbidden: unable to create new content in namespace openshift-template-service-broker because it is being terminated.", "Error from server (Forbidden): serviceaccounts \"apiserver\" is forbidden: unable to create new content in namespace openshift-template-service-broker because it is being terminated.", "Error from server (Forbidden): services \"apiserver\" is forbidden: unable to create new content in namespace openshift-template-service-broker because it is being terminated.", "Error from server (Forbidden): serviceaccounts \"templateservicebroker-client\" is forbidden: unable to create new content in namespace openshift-template-service-broker because it is being terminated.", "Error from server (Forbidden): secrets \"templateservicebroker-client\" is forbidden: unable to create new content in namespace openshift-template-service-broker because it is being terminated."], "stdout": "", "stdout_lines": []}

Version-Release number of selected component (if applicable):
openshift-ansible: 3.9.4-1

How reproducible:
always

Steps to Reproduce:
1.Set up a 3.7 env with service catalog enabled
2.Upgrade to 3.9
3.

Actual results:
Failed to upgrade for unable to create new content in namespace openshift-template-service-broker because it is being terminated.

Expected results:
should upgrade to 3.9

Additional info:
This should be introduced by this pr: https://github.com/openshift/openshift-ansible/pull/7433 which tries to fix bug https://bugzilla.redhat.com/show_bug.cgi?id=1540521

Comment 1 Wenjing Zheng 2018-03-09 08:35:27 UTC
This may due to during project openshift-template-service-broker being deleting, new deployment happens in the same time. So if there is a validation to check whether the project has been deleted before new deploy here: https://github.com/openshift/openshift-ansible/blob/master/roles/template_service_broker/tasks/upgrade.yml, should no this issue.

Comment 2 Michael Gugino 2018-03-09 16:05:23 UTC
Reverting the upgrade deletion step.

PR Created: https://github.com/openshift/openshift-ansible/pull/7474

Comment 3 Scott Dodson 2018-03-09 20:50:04 UTC
release-3.9 backport https://github.com/openshift/openshift-ansible/pull/7480

Comment 5 Wenjing Zheng 2018-03-12 07:54:03 UTC
Verified with openshift-ansible:3.9.7-1, no such error when upgrade now.


Note You need to log in before you can comment on or make changes to this bug.