Bug 1579194
| Summary: | Failed to provision Serviceinstance when met 409 error | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | XiuJuan Wang <xiuwang> |
| Component: | Service Broker | Assignee: | jkim |
| Status: | CLOSED ERRATA | QA Contact: | XiuJuan Wang <xiuwang> |
| Severity: | low | Docs Contact: | |
| Priority: | medium | ||
| Version: | 3.10.0 | CC: | aos-bugs, christian.borba, dymurray, jmatthew, mvardhan, rkshirsa, xiuwang |
| Target Milestone: | --- | Flags: | xiuwang:
needinfo-
|
| Target Release: | 3.11.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | No Doc Update | |
| Doc Text: |
undefined
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-10-11 07:20:00 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
XiuJuan Wang
2018-05-17 06:59:02 UTC
The reproduce step； Add a custom template to openshift project. The template-service-broker clusterservicebroker retrievals catalog every 15 mins. So wait 15 mins, then new adding template will be list in clusterserviceclass. Refresh webconsole, then provision any clusterserviceclass. The serviceinstance will fail from cli side due to comment #0 error. The reproduce percent is 60% https://github.com/automationbroker/bundle-lib/pull/90 This will return an error, and will still cause the broker pod to fail with an 'unknown registry type' error message. (In reply to jkim from comment #2) > https://github.com/automationbroker/bundle-lib/pull/90 > > This will return an error, and will still cause the broker pod to fail with > an 'unknown registry type' error message. invalid comment.. wrong bz Sorry, I can't reproduce this error for adding custom template, but I met the bug error for below steps: Provision a language template with a unexisted repo, the generated serviceinstance will Orphan Mitigation operation always. After have tried many times,the serviceinstance will failed with the bug error. Warning ErrorWithParameters 1h (x2 over 1h) service-catalog-controller-manager failed to prepare parameters nil: secrets "rails-postgresql-example-parameters1vorx" not found Normal Deprovisioning 1h (x4 over 1h) service-catalog-controller-manager The instance is being deprovisioned asynchronously Normal Provisioning 1h (x5 over 1h) service-catalog-controller-manager The instance is being provisioned asynchronously Warning StartingInstanceOrphanMitigation 1h (x5 over 1h) service-catalog-controller-manager The instance provision call failed with an ambiguous error; attempting to deprovision the instance in order to mitigate an orphaned resource Normal OrphanMitigationSuccessful 56m (x18 over 1h) service-catalog-controller-manager Orphan mitigation was completed successfully Warning ProvisionCallFailed 52m (x27 over 1h) service-catalog-controller-manager Provision call failed: Readiness failed on BuildConfig test/rails-postgresql-example Warning ProvisionCallFailed 1m (x161 over 48m) service-catalog-controller-manager Error provisioning ServiceInstance of ClusterServiceClass (K8S: "50be517c-622b-11e8-9888-0e89774d57d8" ExternalName: "rails-postgresql-example") at ClusterServiceBroker "template-service-broker": Status: 409; ErrorMessage: <nil>; Description: <nil>; ResponseError: <nil> (In reply to XiuJuan Wang from comment #5) > Sorry, I can't reproduce this error for adding custom template, but I met > the bug error for below steps: > Provision a language template with a unexisted repo, the generated > serviceinstance will Orphan Mitigation operation always. After have tried > many times,the serviceinstance will failed with the bug error. > > Warning ErrorWithParameters 1h (x2 over 1h) > service-catalog-controller-manager failed to prepare parameters nil: > secrets "rails-postgresql-example-parameters1vorx" not found > Normal Deprovisioning 1h (x4 over 1h) > service-catalog-controller-manager The instance is being deprovisioned > asynchronously > Normal Provisioning 1h (x5 over 1h) > service-catalog-controller-manager The instance is being provisioned > asynchronously > Warning StartingInstanceOrphanMitigation 1h (x5 over 1h) > service-catalog-controller-manager The instance provision call failed with > an ambiguous error; attempting to deprovision the instance in order to > mitigate an orphaned resource > Normal OrphanMitigationSuccessful 56m (x18 over 1h) > service-catalog-controller-manager Orphan mitigation was completed > successfully > Warning ProvisionCallFailed 52m (x27 over 1h) > service-catalog-controller-manager Provision call failed: Readiness failed > on BuildConfig test/rails-postgresql-example > Warning ProvisionCallFailed 1m (x161 over 48m) > service-catalog-controller-manager Error provisioning ServiceInstance of > ClusterServiceClass (K8S: "50be517c-622b-11e8-9888-0e89774d57d8" > ExternalName: "rails-postgresql-example") at ClusterServiceBroker > "template-service-broker": Status: 409; ErrorMessage: <nil>; Description: > <nil>; ResponseError: <nil> Thank you. From the title and the description, I though this bug occurs only when a custom template was provisioned. However, your last comment suggests that you see this when you have an invalid parameters and force the provision to fail, is that correct? Assuming that you only see this on a bad provisions, when I purposefully made the provision fail, via invalid parameters (e.g. language template provisioned with an invalid git repo), I was able to finally reproduce the similar “Status 409 Error” message in the serviceinstance, after trying numerous times. From what I’ve investigated so far, this may not be a template service broker bug. The TSB logs show that the provision failed, and a proper last-operation message was sent to the service catalog. However, as you have already pointed out, after the provision/deprovision is ‘tried many times’ by the orphan mitigation, I get the “status 409 Error” message. I will need to investigate this further, but it appears that the provision/deprovision is functioning correctly from the Template service broker. But the orphan mitigation from the service-catalog may be calling the provision twice in a row somehow. And since the template’s secrets already exist from the previous provision, the second provision w/o a deprovision will throw the “status 409 error” message. Realigning this to 3.11 as we investigate further (In reply to jkim from comment #6) > Thank you. From the title and the description, I though this bug occurs > only when a custom template was provisioned. However, your last comment > suggests that you see this when you have an invalid parameters and force the > provision to fail, is that correct? Correct, That's what I do to reproduce this bug stablely In 3.10, serviceinstance will be marked as failed from cli side when met 409 error during provision processing, but keeping status shown in webside. However the 409 error is not a issue to block serviceinstance status in ocp 3.9. Comment #8 is not the only way to reproduce this bug. Even met 409 error with a normal privision process with 50% probability. Commits pushed to master at https://github.com/openshift/origin https://github.com/openshift/origin/commit/f446296dbc24cdd9b3a002bee479d7ada828a261 Bug 1579194 - Fixes handling of multiple provision request https://github.com/openshift/origin/commit/ec4fd653d1f58e6c6c50b532a3eb39b42e8ae844 Merge pull request #20099 from johnkim76/bug_1579194 Bug 1579194 - Fixes handling of multiple provision request Recently added to advisory: openshift-enterprise-asb-container-v3.11.0-0.20.0.0 openshift-enterprise-apb-tools-container-v3.11.0-0.20.0.0 This should pick up the bundle-lib changes. Waiting on origin to move this to ON_QA. Can't reproduce this bug. Tried servel ways mentioned in the bugs, don't met 409 error any more. openshift v3.11.0-0.21.0 kubernetes v1.11.0+d4cacc0 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:2652 *** Bug 1551988 has been marked as a duplicate of this bug. *** *** Bug 1637831 has been marked as a duplicate of this bug. *** |