Bug 1585649

Summary: User project cannot be deleted while mediawiki provision failed
Product: OpenShift Container Platform Reporter: Zhang Cheng <chezhang>
Component: Service BrokerAssignee: Jason Montleon <jmontleo>
Status: CLOSED ERRATA QA Contact: Zhang Cheng <chezhang>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.10.0CC: aos-bugs, jiazha, jmatthew, shurley, zhsun, zitang
Target Milestone: ---   
Target Release: 3.10.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
undefined
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-07-30 19:16:54 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Zhang Cheng 2018-06-04 09:47:07 UTC
Description of problem: 
User project cannot be deleted while mediawiki provision failed


service-catalog & asb image using images:
# service-catalog --version
v3.10.0-0.58.0;Upstream:v0.1.19
# asbd --version
1.2.16
Mediawiki-apb from stage registry: openshift-enterprise-mediawiki-apb-v3.10.0-0.54.0.1


How reproducible:
Always


Steps to Reproduce:
1. Provison a mediawiki apb with same username and passwd in project "test123" (provision should failed by expect) from web console.
2. Delete user project "test123" directly.


Actual results:  
1. Provision failed and auto trigger deprovision sandbox by desired.
2. Cannot delete user project "test123"
# oc describe serviceinstance -n test123
Name:         rhcc-mediawiki-apb-rcknr
Namespace:    test123
Labels:       <none>
Annotations:  <none>
API Version:  servicecatalog.k8s.io/v1beta1
Kind:         ServiceInstance
Metadata:
  Creation Timestamp:             2018-06-04T09:32:32Z
  Deletion Grace Period Seconds:  0
  Deletion Timestamp:             2018-06-04T09:43:03Z
  Finalizers:
    kubernetes-incubator/service-catalog
  Generate Name:     rhcc-mediawiki-apb-
  Generation:        2
  Resource Version:  60778
  Self Link:         /apis/servicecatalog.k8s.io/v1beta1/namespaces/test123/serviceinstances/rhcc-mediawiki-apb-rcknr
  UID:               34f1af98-67da-11e8-99f7-0a580a800008
Spec:
  Cluster Service Class External Name:  rhcc-mediawiki-apb
  Cluster Service Class Ref:
    Name:                              afead97b16e87b703986f87c20ec392c
  Cluster Service Plan External Name:  default
  Cluster Service Plan Ref:
    Name:       97a9d79595adeb3cdda2b64fe0e6af2c
  External ID:  34f1aece-67da-11e8-99f7-0a580a800008
  Parameters From:
    Secret Key Ref:
      Key:          parameters
      Name:         rhcc-mediawiki-apb-parameters7pror
  Update Requests:  0
  User Info:
    Groups:
      system:serviceaccounts
      system:serviceaccounts:kube-system
      system:authenticated
    UID:       
    Username:  system:serviceaccount:kube-system:namespace-controller
Status:
  Async Op In Progress:  true
  Conditions:
    Last Transition Time:  2018-06-04T09:32:32Z
    Message:               The instance is being deprovisioned asynchronously (0%: Starting)
    Reason:                Deprovisioning
    Status:                False
    Type:                  Ready
    Last Transition Time:  2018-06-04T09:32:41Z
    Message:               Provision call failed: Error occurred during provision. Please contact administrator if the issue persists.
    Reason:                ProvisionCallFailed
    Status:                True
    Type:                  OrphanMitigation
  Current Operation:       Provision
  Deprovision Status:      Required
  In Progress Properties:
    Cluster Service Plan External ID:    97a9d79595adeb3cdda2b64fe0e6af2c
    Cluster Service Plan External Name:  default
    Parameter Checksum:                  fc314589c54aab15cc3b4a72f02ac4f954238b991f056d51dac8e8a656e6033d
    Parameters:
      Mediawiki _ Admin _ Pass:   <redacted>
      Mediawiki _ Admin _ User:   <redacted>
      Mediawiki _ Db _ Schema:    <redacted>
      Mediawiki _ Site _ Lang:    <redacted>
      Mediawiki _ Site _ Name:    <redacted>
      Mediawiki _ Volume _ Size:  <redacted>
    User Info:
      Extra:
        Scopes . Authorization . Openshift . Io:
          user:full
      Groups:
        system:authenticated:oauth
        system:authenticated
      UID:                        
      Username:                   chezhang
  Last Operation:                 6ff26c5f-2af0-44b0-a9e3-67a1dfb18262
  Observed Generation:            1
  Operation Start Time:           2018-06-04T09:32:33Z
  Orphan Mitigation In Progress:  true
  Provision Status:               
  Reconciled Generation:          0
Events:                           <none>


Expected results: 
2. User project "test123" should be deleted succeed.


Addition info: 
None.

Comment 2 Jason Montleon 2018-06-04 14:33:07 UTC
At least part of this is a missing conditional on the mediawiki-apb.
https://github.com/ansibleplaybookbundle/mediawiki-apb/pull/35

With this the provision fails as expected and the deprovision succeeds as expected.

However, I still see the instance state as "The service was marked for deletion." and the namespace is not being deleted.

Comment 3 Jason Montleon 2018-06-04 15:11:39 UTC
We might be better off documenting that if the username and password are the same the user will be unable to login, and add a description on the field in the apb to hopefully prevent users from doing this.

I've confirmed that the provision/deprovision and namespace deletion work with the below PR.

https://github.com/ansibleplaybookbundle/mediawiki-apb/pull/36

Comment 4 Shawn Hurley 2018-06-04 18:37:55 UTC
This seems to be an issue with the messages that the broker is receiving and the order in which they happen.

It appears the broker gets a Failed Pod messages and marks the status as failed.
But then it gets a last operation (starting 0%...) from the pod. This then sets the status as in progress. What happens then is the Service Instance is waiting for an update that will never happen and this will cause the finalizer to stop the namespace from being deleted. 

I am working on a PR to fix this now.

Comment 5 David Zager 2018-06-04 20:16:20 UTC
The two PRs to address this bug:

- https://github.com/ansibleplaybookbundle/mediawiki-apb/pull/35 allows you to deprovision mediawiki when username|password are the same
- https://github.com/openshift/ansible-service-broker/pull/976 addresses message handling

Comment 7 Zhang Cheng 2018-06-06 02:45:40 UTC
Tested with asb 1.2.17 + openshift-enterprise-mediawiki-apb-v3.10.0-0.54.0.1(from stage). The user project can be deleted succeed. 

But, deprovision still failed since image mediawiki-apb not update in downstream registry(stage).

Refer to Jason's comments:
At least part of this is a missing conditional on the mediawiki-apb.
https://github.com/ansibleplaybookbundle/mediawiki-apb/pull/35

With this the provision fails as expected and the deprovision succeeds as expected.


I'm changing status to Modified since no downstream image ready for test. Thanks.

Comment 10 Zhang Cheng 2018-06-07 06:03:49 UTC
Verified and Passed with asb 1.2.17 + openshift-enterprise-mediawiki-container-v3.10.0-0.63.0.0

Deprovision succeed and user project can be deleted.


# oc logs apb-ce04e0f6-0000-44f4-bd5e-efdfc77dfbac

PLAY [Deprovision mediawiki-apb from openshift] ********************************

TASK [ansibleplaybookbundle.asb-modules : debug] *******************************
skipping: [localhost]

TASK [mediawiki-apb : Update last operation] ***********************************
changed: [localhost]

TASK [mediawiki-apb : Determine cluster type] **********************************
changed: [localhost]

TASK [mediawiki-apb : Set facts] ***********************************************
ok: [localhost]

TASK [mediawiki-apb : Include variables based on ansible version] **************
skipping: [localhost]

TASK [mediawiki-apb : Validate Admin User and Password] ************************
skipping: [localhost]

TASK [mediawiki-apb : Set mediawiki objects state=absent] **********************
ok: [localhost] => (item=service.yaml)
ok: [localhost] => (item=pvc.yaml)

TASK [mediawiki-apb : Set mediawiki route state=absent] ************************
ok: [localhost]

TASK [mediawiki-apb : Set mediawiki deployment state=absent] *******************
ok: [localhost]

TASK [mediawiki-apb : Update last operation] ***********************************
changed: [localhost]

PLAY RECAP *********************************************************************
localhost                  : ok=7    changed=3    unreachable=0    failed=0

Comment 12 errata-xmlrpc 2018-07-30 19:16:54 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1816