Description of problem: Update database APB to a nonexistent version, then ServiceInstance can't recover from the invalid version update. Version-Release number of selected component (if applicable): openshift v3.9.0-0.23.0 kubernetes v1.9.1+a0ce1bc657 etcd 3.2.8 How reproducible: Always Steps to Reproduce: 1. Provision a APB (MySQL for example) 2. Update its version to a nonexistent one (5.6 -> 6) 3. Check pod and serviceinstance. 4. Update to a valid version (5.6 or 5.7). 5. Actual results: 3. Because there is no mysql:v6 to pull [root@host-172-16-120-114 ~]# oc get pod NAME READY STATUS RESTARTS AGE mysql-5.6-dev-1-bhwt2 1/1 Running 0 10m mysql-6-dev-1-42lvx 0/1 ImagePullBackOff 0 3m mysql-6-dev-1-deploy 1/1 Running 0 3m 4. # Please edit the object below. Lines beginning with a '#' will be ignored, # and an empty file will abort the edit. If an error occurs while saving this file will be # reopened with the relevant failures. # # serviceinstances "rh-mysql-apb-lsnx8" was not valid: # * spec: Forbidden: Another update for this service instance is in progress # [root@host-172-16-120-114 ~]# oc edit serviceinstance rh-mysql-apb-lsnx8 error: serviceinstances "rh-mysql-apb-lsnx8" is invalid A copy of your changes has been stored to "/tmp/oc-edit-8dcpd.yaml" error: Edit cancelled, no valid changes were saved. 5. [root@host-172-16-120-114 ~]# oc describe serviceinstance rh-mysql-apb-lsnx8 | grep -A10 Events Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning ErrorWithParameters 21m service-catalog-controller-manager failed to prepare parameters nil: secrets "rh-mysql-apb-parameters5cja4" not found Normal Provisioning 21m service-catalog-controller-manager The instance is being provisioned asynchronously Normal ProvisionedSuccessfully 21m service-catalog-controller-manager The instance was provisioned successfully Normal UpdatingInstance 13m service-catalog-controller-manager The instance is being updated asynchronously Warning UpdateInstanceCallFailed 5m service-catalog-controller-manager Update call failed: Error occurred during update. Please contact administrator if it persists. Expected results: 4. ServiceInstance should be edit and new pod runs with an effective version Additional info:
Can you tell me how you updated the version to 6?
(In reply to Jason Montleon from comment #1) > Can you tell me how you updated the version to 6? Hi, version and other information are encoded with base64 and stored in the secret，so I changed its secret to update the version. Note: Step 1 and 2 can be skipped. Get the decoded parameters from web UI->secret->reveal secret 1. # oc get secret rh-mariadb-apb-parametersxxx -o yaml | grep parameters parameters: eyJteXNxbF9kYXRhYmFzZSI6ImRldmVsIiwibXlzcWxfdXNlciI6ImRldmVsIiwibXlzcWxfdmVyc2lvbiI6IjUuNiIsInNlcnZpY2VfbmFtZSI6Im15c3FsIiwibXlzcWxfcGFzc3dvcmQiOiIxMTExMTEifQ== 2. Decode parameters # echo -n 'eyJteXNxbF9kYXRhYmFzZSI6ImRldmVsIiwibXlzcWxfdXNlciI6ImRldmVsIiwibXlzcWxfdmVyc2lvbiI6IjUuNiIsInNlcnZpY2VfbmFtZSI6Im15c3FsIiwibXlzcWxfcGFzc3dvcmQiOiIxMTExMTEifQ==' | base64 -d {"mysql_database":"devel","mysql_user":"devel","mysql_version":"5.6","service_name":"mysql","mysql_password":"111111"} 3. Encode parameters with a new mysql_version (or use online base64 tool) # echo -n '{"mysql_database":"devel","mysql_user":"devel","mysql_version":"6","service_name":"mysql","mysql_password":"111111"}' | base64 eyJteXNxbF9kYXRhYmFzZSI6ImRldmVsIiwibXlzcWxfdXNlciI6ImRldmVsIiwibXlzcWxfdmVy c2lvbiI6IjYiLCJzZXJ2aWNlX25hbWUiOiJteXNxbCIsIm15c3FsX3Bhc3N3b3JkIjoiMTExMTEx In0= 4. Use this new encoded one to update the secret pamaters # oc edit secret rh-mariadb-apb-parametersxxx 5. Increase spec.updateRequests by 1 each time once secret is updated # oc edit serviceinstance <mysql_serviceinstance>
https://github.com/openshift/ansible-service-broker/pull/713 Incrementing the UpdateRequest without making any valid changes can also produce an identical deployment config during update so we need to take care not to delete it in those cases. https://github.com/ansibleplaybookbundle/postgresql-apb/pull/31 https://github.com/ansibleplaybookbundle/mariadb-apb/pull/22 https://github.com/ansibleplaybookbundle/mysql-apb/pull/22
Asb image is 1.1.9 and ready for test , change to ON_QA.
Verified. ASB : 1.1.9 ; brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/ose-ansible-service-broker:v3.9 Service-catalog : 0.1.3 brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/openshift3/ose-service-catalog:v3.9 Downstream image. update postgresql apb version 9.7->9.9 ->9.5 , succeed.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:0489