Red Hat Bugzilla – Bug 1579194
Failed to provision Serviceinstance when met 409 error
Last modified: 2018-10-11 03:20:21 EDT
Description of problem: Serviceinstance goes to failed after templateinstance ready, met below error: Error provisioning ServiceInstance of ClusterServiceClass (K8S: "02f28a4b-597e-11e8-82fb-0eaf4f6d5fd2" ExternalName: "mariadb-ephemeral") at ClusterServiceBroker "template-service-broker": Status: 409 Version-Release number of selected component (if applicable): Server https://***:8443 openshift v3.10.0-0.47.0 kubernetes v1.10.0+b81c8f8 How reproducible: always Steps to Reproduce: 1.Privision a clusterserviceclass from browse catalog 2.Check templateinstance status 3.Check serviceinstance status Actual results: 2.Templateinstance goes to ready 3. # oc describe serviceinstance mariadb-ephemeral-s4nn2 -n test Name: mariadb-ephemeral-s4nn2 Namespace: test Labels: <none> Annotations: <none> API Version: servicecatalog.k8s.io/v1beta1 Kind: ServiceInstance Metadata: Creation Timestamp: 2018-05-17T06:29:26Z Finalizers: kubernetes-incubator/service-catalog Generate Name: mariadb-ephemeral- Generation: 1 Resource Version: 28790 Self Link: /apis/servicecatalog.k8s.io/v1beta1/namespaces/test/serviceinstances/mariadb-ephemeral-s4nn2 UID: a550d757-599b-11e8-9592-0a580a800005 Spec: Cluster Service Class External Name: mariadb-ephemeral Cluster Service Class Ref: Name: 02f28a4b-597e-11e8-82fb-0eaf4f6d5fd2 Cluster Service Plan External Name: default Cluster Service Plan Ref: Name: 02f28a4b-597e-11e8-82fb-0eaf4f6d5fd2 External ID: a550d3dc-599b-11e8-9592-0a580a800005 Parameters From: Secret Key Ref: Key: parameters Name: mariadb-ephemeral-parametersr9nyi Update Requests: 0 User Info: Extra: Scopes . Authorization . Openshift . Io: user:full Groups: system:authenticated:oauth system:authenticated UID: Username: xiuwang Status: Async Op In Progress: false Conditions: Last Transition Time: 2018-05-17T06:29:26Z Message: Error provisioning ServiceInstance of ClusterServiceClass (K8S: "02f28a4b-597e-11e8-82fb-0eaf4f6d5fd2" ExternalName: "mariadb-ephemeral") at ClusterServiceBroker "template-service-broker": Status: 409; ErrorMessage: <nil>; Description: <nil>; ResponseError: <nil> Reason: ProvisionCallFailed Status: False Type: Ready Current Operation: Provision Deprovision Status: NotRequired In Progress Properties: Cluster Service Plan External ID: 02f28a4b-597e-11e8-82fb-0eaf4f6d5fd2 Cluster Service Plan External Name: default Parameter Checksum: f35ee9ba5bfeebfc739427b72f8404facde64e8190cfe5f5f5442ea991c9b1f2 Parameters: DATABASE _ SERVICE _ NAME: <redacted> MARIADB _ VERSION: <redacted> MEMORY _ LIMIT: <redacted> MYSQL _ DATABASE: <redacted> NAMESPACE: <redacted> User Info: Extra: Scopes . Authorization . Openshift . Io: user:full Groups: system:authenticated:oauth system:authenticated UID: Username: xiuwang Observed Generation: 1 Operation Start Time: 2018-05-17T06:29:27Z Orphan Mitigation In Progress: false Provision Status: Reconciled Generation: 0 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning ErrorWithParameters 5m (x2 over 5m) service-catalog-controller-manager failed to prepare parameters nil: secrets "mariadb-ephemeral-parametersr9nyi" not found Normal Provisioning 5m service-catalog-controller-manager The instance is being provisioned asynchronously Warning ProvisionCallFailed 56s (x30 over 5m) service-catalog-controller-manager Error provisioning ServiceInstance of ClusterServiceClass (K8S: "02f28a4b-597e-11e8-82fb-0eaf4f6d5fd2" ExternalName: "mariadb-ephemeral") at ClusterServiceBroker "template-service-broker": Status: 409; ErrorMessage: <nil>; Description: <nil>; ResponseError: <nil> Expected results: Serviceinstance should be ready Additional info: This error existed in v3.9 and v3.7 cluster too, but won't conduce serviceinstance failed
The reproduce step； Add a custom template to openshift project. The template-service-broker clusterservicebroker retrievals catalog every 15 mins. So wait 15 mins, then new adding template will be list in clusterserviceclass. Refresh webconsole, then provision any clusterserviceclass. The serviceinstance will fail from cli side due to comment #0 error. The reproduce percent is 60%
https://github.com/automationbroker/bundle-lib/pull/90 This will return an error, and will still cause the broker pod to fail with an 'unknown registry type' error message.
(In reply to jkim from comment #2) > https://github.com/automationbroker/bundle-lib/pull/90 > > This will return an error, and will still cause the broker pod to fail with > an 'unknown registry type' error message. invalid comment.. wrong bz
Sorry, I can't reproduce this error for adding custom template, but I met the bug error for below steps: Provision a language template with a unexisted repo, the generated serviceinstance will Orphan Mitigation operation always. After have tried many times,the serviceinstance will failed with the bug error. Warning ErrorWithParameters 1h (x2 over 1h) service-catalog-controller-manager failed to prepare parameters nil: secrets "rails-postgresql-example-parameters1vorx" not found Normal Deprovisioning 1h (x4 over 1h) service-catalog-controller-manager The instance is being deprovisioned asynchronously Normal Provisioning 1h (x5 over 1h) service-catalog-controller-manager The instance is being provisioned asynchronously Warning StartingInstanceOrphanMitigation 1h (x5 over 1h) service-catalog-controller-manager The instance provision call failed with an ambiguous error; attempting to deprovision the instance in order to mitigate an orphaned resource Normal OrphanMitigationSuccessful 56m (x18 over 1h) service-catalog-controller-manager Orphan mitigation was completed successfully Warning ProvisionCallFailed 52m (x27 over 1h) service-catalog-controller-manager Provision call failed: Readiness failed on BuildConfig test/rails-postgresql-example Warning ProvisionCallFailed 1m (x161 over 48m) service-catalog-controller-manager Error provisioning ServiceInstance of ClusterServiceClass (K8S: "50be517c-622b-11e8-9888-0e89774d57d8" ExternalName: "rails-postgresql-example") at ClusterServiceBroker "template-service-broker": Status: 409; ErrorMessage: <nil>; Description: <nil>; ResponseError: <nil>
(In reply to XiuJuan Wang from comment #5) > Sorry, I can't reproduce this error for adding custom template, but I met > the bug error for below steps: > Provision a language template with a unexisted repo, the generated > serviceinstance will Orphan Mitigation operation always. After have tried > many times,the serviceinstance will failed with the bug error. > > Warning ErrorWithParameters 1h (x2 over 1h) > service-catalog-controller-manager failed to prepare parameters nil: > secrets "rails-postgresql-example-parameters1vorx" not found > Normal Deprovisioning 1h (x4 over 1h) > service-catalog-controller-manager The instance is being deprovisioned > asynchronously > Normal Provisioning 1h (x5 over 1h) > service-catalog-controller-manager The instance is being provisioned > asynchronously > Warning StartingInstanceOrphanMitigation 1h (x5 over 1h) > service-catalog-controller-manager The instance provision call failed with > an ambiguous error; attempting to deprovision the instance in order to > mitigate an orphaned resource > Normal OrphanMitigationSuccessful 56m (x18 over 1h) > service-catalog-controller-manager Orphan mitigation was completed > successfully > Warning ProvisionCallFailed 52m (x27 over 1h) > service-catalog-controller-manager Provision call failed: Readiness failed > on BuildConfig test/rails-postgresql-example > Warning ProvisionCallFailed 1m (x161 over 48m) > service-catalog-controller-manager Error provisioning ServiceInstance of > ClusterServiceClass (K8S: "50be517c-622b-11e8-9888-0e89774d57d8" > ExternalName: "rails-postgresql-example") at ClusterServiceBroker > "template-service-broker": Status: 409; ErrorMessage: <nil>; Description: > <nil>; ResponseError: <nil> Thank you. From the title and the description, I though this bug occurs only when a custom template was provisioned. However, your last comment suggests that you see this when you have an invalid parameters and force the provision to fail, is that correct? Assuming that you only see this on a bad provisions, when I purposefully made the provision fail, via invalid parameters (e.g. language template provisioned with an invalid git repo), I was able to finally reproduce the similar “Status 409 Error” message in the serviceinstance, after trying numerous times. From what I’ve investigated so far, this may not be a template service broker bug. The TSB logs show that the provision failed, and a proper last-operation message was sent to the service catalog. However, as you have already pointed out, after the provision/deprovision is ‘tried many times’ by the orphan mitigation, I get the “status 409 Error” message. I will need to investigate this further, but it appears that the provision/deprovision is functioning correctly from the Template service broker. But the orphan mitigation from the service-catalog may be calling the provision twice in a row somehow. And since the template’s secrets already exist from the previous provision, the second provision w/o a deprovision will throw the “status 409 error” message.
Realigning this to 3.11 as we investigate further
(In reply to jkim from comment #6) > Thank you. From the title and the description, I though this bug occurs > only when a custom template was provisioned. However, your last comment > suggests that you see this when you have an invalid parameters and force the > provision to fail, is that correct? Correct, That's what I do to reproduce this bug stablely
In 3.10, serviceinstance will be marked as failed from cli side when met 409 error during provision processing, but keeping status shown in webside. However the 409 error is not a issue to block serviceinstance status in ocp 3.9. Comment #8 is not the only way to reproduce this bug. Even met 409 error with a normal privision process with 50% probability.
https://github.com/openshift/origin/pull/20099
Commits pushed to master at https://github.com/openshift/origin https://github.com/openshift/origin/commit/f446296dbc24cdd9b3a002bee479d7ada828a261 Bug 1579194 - Fixes handling of multiple provision request https://github.com/openshift/origin/commit/ec4fd653d1f58e6c6c50b532a3eb39b42e8ae844 Merge pull request #20099 from johnkim76/bug_1579194 Bug 1579194 - Fixes handling of multiple provision request
Recently added to advisory: openshift-enterprise-asb-container-v3.11.0-0.20.0.0 openshift-enterprise-apb-tools-container-v3.11.0-0.20.0.0 This should pick up the bundle-lib changes. Waiting on origin to move this to ON_QA.
Can't reproduce this bug. Tried servel ways mentioned in the bugs, don't met 409 error any more. openshift v3.11.0-0.21.0 kubernetes v1.11.0+d4cacc0
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2018:2652