Description of problem: When provision mediawiki apb, falied with "Something went wrong creating project mediawiki-project!" Version-Release number of the following components: openshift v3.6.136 kubernetes v1.6.1+5115d708d7 etcd 3.2.1 openshift-ansible-3.6.140-1.git.0.4a02427.el7.noarch.rpm How reproducible: Always Steps to Reproduce: 1. When provision mediawiki [root@ip-172-18-6-17 asb]# cat provison.sh #!/bin/bash -e instanceUUID="5d85082a-c8a9-49e3-82a1-3b6ebe6be644" planUUID="4c10ff42-be89-420a-9bab-27a9bef9aed8" serviceUUID="4fbcc051-03af-40c7-86e2-dc12977e6b5d" req="{ \"plan_id\": \"$planUUID\", \"service_id\": \"$serviceUUID\", \"context\": { \"platform\": \"kubernetes\", \"namespace\": \"mediawiki-project\" }, \"parameters\": { \"mediawiki_db_schema\": \"mediawiki\", \"mediawiki_site_name\": \"mediawiki\", \"mediawiki_site_lang\": \"en\", \"mediawiki_admin_user\": \"admin\", \"mediawiki_admin_pass\":\"redhat\" } }" curl \ -X PUT \ -H 'X-Broker-API-Version: 2.9' \ -H 'Content-Type: application/json' \ -d "$req" \ -v \ "http://172.30.112.253:1338/v2/service_instances/$instanceUUID?accepts_incomplete=true" 2. error logs from asb pod: [2017-07-11T10:57:20.833Z] [INFO] ASYNC provisioning in progress [2017-07-11T10:57:20.833Z] [NOTICE] ============================================================ [2017-07-11T10:57:20.833Z] [NOTICE] PROVISIONING [2017-07-11T10:57:20.833Z] [NOTICE] ============================================================ [2017-07-11T10:57:20.833Z] [NOTICE] Spec.ID: 4fbcc051-03af-40c7-86e2-dc12977e6b5d [2017-07-11T10:57:20.833Z] [NOTICE] Spec.Name: mediawiki-apb [2017-07-11T10:57:20.833Z] [NOTICE] Spec.Image: openshift3/mediawiki-apb [2017-07-11T10:57:20.833Z] [NOTICE] Spec.Description: Mediawiki123 apb implementation [2017-07-11T10:57:20.836Z] [NOTICE] ============================================================ [2017-07-11T10:57:20.836Z] [INFO] Checking if project mediawiki-project exists... 10.129.0.1 - - [11/Jul/2017:10:57:20 +0000] "PUT /v2/service_instances/bdeb07ca-2da8-4742-9312-561c6ec86ae0?accepts_incomplete=true HTTP/1.1" 202 58 [2017-07-11T10:57:21.229Z] [INFO] Project mediawiki-project does NOT exist, creating project... [2017-07-11T10:57:21.579Z] [ERROR] Something went wrong creating project mediawiki-project! [2017-07-11T10:57:21.579Z] [ERROR] exit status 1 [2017-07-11T10:57:21.579Z] [ERROR] broker::Provision error occurred. [2017-07-11T10:57:21.579Z] [ERROR] exit status 1 [2017-07-11T10:57:21.579Z] [ERROR] Attempting to destroy APB sandbox if it has been created [2017-07-11T10:57:21.579Z] [INFO] Requested destruction of APB sandbox with empty handle, skipping. Actual results: Expected results: Additional info:
DeShuai, The project needs to exist prior to invoking provision. The APB is not expected to create the project at this point. (We can consider allowing project creation post 3.6.0, for 3.6.0 we decided project would exist prior to APB) Please create the project and then re-test
The error seems to be from `oc new-project mediawiki-project`. Can you run that by hand and paste the output?
Eventhrough there is already exist the project it still failed. After further debug, The issue here is the service account has no permission get/create project. we don't need create the project before provision. If no project, it will be created. After add clusteRrole 'cluster-admin'(actually we don't need so high role) the the sa 'asb', it can provision. [root@ip-172-18-6-17 asb]# oadm policy add-cluster-role-to-user cluster-admin system:serviceaccount:openshift-ansible-service-broker:asb cluster role "cluster-admin" added: "system:serviceaccount:openshift-ansible-service-broker:asb" [2017-07-11T15:01:36.086Z] [INFO] ASYNC provisioning in progress [2017-07-11T15:01:36.086Z] [NOTICE] ============================================================ [2017-07-11T15:01:36.086Z] [NOTICE] PROVISIONING [2017-07-11T15:01:36.086Z] [NOTICE] ============================================================ [2017-07-11T15:01:36.086Z] [NOTICE] Spec.Id: 4fbcc051-03af-40c7-86e2-dc12977e6b5d [2017-07-11T15:01:36.086Z] [NOTICE] Spec.Name: mediawiki-apb [2017-07-11T15:01:36.086Z] [NOTICE] Spec.Image: openshift3/mediawiki-apb [2017-07-11T15:01:36.086Z] [NOTICE] Spec.Description: Mediawiki123 apb implementation [2017-07-11T15:01:36.086Z] [NOTICE] ============================================================ [2017-07-11T15:01:36.086Z] [INFO] Checking if project mediawiki-project exists... 10.129.0.1 - - [11/Jul/2017:15:01:36 +0000] "PUT /v2/service_instances/5e1745fa-3025-44e8-8998-acce0557caa8?accepts_incomplete=true HTTP/1.1" 202 58 [2017-07-11T15:01:36.542Z] [INFO] Project mediawiki-project does NOT exist, creating project... [2017-07-11T15:01:37.586Z] [INFO] Successfully created project mediawiki-project [2017-07-11T15:01:37.586Z] [DEBUG] oc new-project output: [2017-07-11T15:01:37.586Z] [DEBUG] Now using project "mediawiki-project" on server "https://172.30.0.1:443". You can add applications to this project with the 'new-app' command. For example, try: oc new-app centos/ruby-22-centos7~https://github.com/openshift/ruby-ex.git to build a new example application in Ruby. [2017-07-11T15:01:37.587Z] [DEBUG] clusterConfig: [2017-07-11T15:01:37.587Z] [DEBUG] name:[ mediawiki-apb ] [2017-07-11T15:01:37.587Z] [DEBUG] image:[ openshift3/mediawiki-apb ] [2017-07-11T15:01:37.587Z] [DEBUG] action:[ provision ] [2017-07-11T15:01:37.587Z] [DEBUG] Creating resource file dir: /tmp/asb-resource-files [2017-07-11T15:01:37.587Z] [DEBUG] Creating resource file /tmp/asb-resource-files/apb-0a3664f9-0e14-45ff-b2e5-6aa18f164aff.yaml [2017-07-11T15:01:37.588Z] [INFO] Successfully wrote resources to /tmp/asb-resource-files/apb-0a3664f9-0e14-45ff-b2e5-6aa18f164aff.yaml [2017-07-11T15:01:37.588Z] [DEBUG] Creating resources from file at path: /tmp/asb-resource-files/apb-0a3664f9-0e14-45ff-b2e5-6aa18f164aff.yaml [2017-07-11T15:01:38.18Z] [DEBUG] Successfully created resources, oc create -f output: [2017-07-11T15:01:38.18Z] [DEBUG] serviceaccount "apb-0a3664f9-0e14-45ff-b2e5-6aa18f164aff" created rolebinding "apb-0a3664f9-0e14-45ff-b2e5-6aa18f164aff" created [2017-07-11T15:01:38.18Z] [INFO] Successfully created apb sandbox: [ apb-0a3664f9-0e14-45ff-b2e5-6aa18f164aff ] [2017-07-11T15:01:38.18Z] [NOTICE] Creating pod "apb-0a3664f9-0e14-45ff-b2e5-6aa18f164aff" in the mediawiki-project namespace [2017-07-11T15:01:39.379Z] [DEBUG] Calling monitorOutput on apb-0a3664f9-0e14-45ff-b2e5-6aa18f164aff [2017-07-11T15:01:39.903Z] [DEBUG] oc log output: Error from server (BadRequest): container "apb" in pod "apb-0a3664f9-0e14-45ff-b2e5-6aa18f164aff" is waiting to start: ContainerCreating [2017-07-11T15:01:45.904Z] [INFO] Container not up yet, retrying 1 of 150 on pod apb-0a3664f9-0e14-45ff-b2e5-6aa18f164aff [2017-07-11T15:01:46.392Z] [DEBUG] oc log output: Error from server (BadRequest): container "apb" in pod "apb-0a3664f9-0e14-45ff-b2e5-6aa18f164aff" is waiting to start: image can't be pulled [2017-07-11T15:01:46.392Z] [NOTICE] WARNING: Unexpected output from apb pod [2017-07-11T15:01:46.392Z] [NOTICE] Will keep retrying, but it's possible something has gone wrong. [2017-07-11T15:01:46.392Z] [NOTICE] image can't be pulled [2017-07-11T15:01:52.392Z] [INFO] Container not up yet, retrying 2 of 150 on pod apb-0a3664f9-0e14-45ff-b2e5-6aa18f164aff [2017-07-11T15:01:52.828Z] [DEBUG] oc log output: Error from server (BadRequest): container "apb" in pod "apb-0a3664f9-0e14-45ff-b2e5-6aa18f164aff" is waiting to start: image can't be pulled [2017-07-11T15:01:52.829Z] [NOTICE] WARNING: Unexpected output from apb pod [2017-07-11T15:01:52.829Z] [NOTICE] Will keep retrying, but it's possible something has gone wrong. [2017-07-11T15:01:52.829Z] [NOTICE] image can't be pulled
sorry mistake set the assign back. correct it.
Looks like we need to add something like this to the installer: oadm policy add-cluster-role-to-user <role> system:serviceaccount:openshift-ansible-service-broker:asb Looking at some of the docs, I think role as basic-user makes sense.
Agreed, we need this change in the installer for 3.6.0 Note the broker will need the ability to create service accounts in other projects than it's own. We have been using 'cluster-admin' for our testing. I don't think basic-user will be sufficient.
tracking here: https://github.com/openshift/openshift-ansible/pull/4736
system:serviceaccount:openshift-ansible-service-broker:asb also need permission to create/delete sa/pod in other project [root@host-8-175-186 dma]# oc logs asb-1222154024-1wkbz Using config file mounted to /etc/ansible-service-broker/config.yaml ============================================================ == Starting Ansible Service Broker... == ============================================================ [2017-07-17T06:44:40.889Z] [INFO] == ETCD CX == [2017-07-17T06:44:40.889Z] [INFO] EtcdHost: etcd [2017-07-17T06:44:40.889Z] [INFO] EtcdPort: 2379 [2017-07-17T06:44:40.889Z] [INFO] Endpoints: [http://etcd:2379] [2017-07-17T06:44:40.898Z] [INFO] Etcd Version [Server: 3.2.3, Cluster: 3.2.0] [2017-07-17T06:44:40.899Z] [INFO] Cluster Client Created [2017-07-17T06:44:40.941Z] [INFO] Kubernetes version: v1.6.1+5115d708d7 [2017-07-17T06:44:40.941Z] [INFO] == REGISTRY CX == [2017-07-17T06:44:40.941Z] [INFO] Name: rhcc [2017-07-17T06:44:40.941Z] [INFO] Url: registry.access.stage.redhat.com [2017-07-17T06:44:40.942Z] [INFO] Listening for provision messages [2017-07-17T06:44:41.265Z] [INFO] Initiating Recovery Process [2017-07-17T06:44:41.268Z] [INFO] Recovery complete [2017-07-17T06:44:41.268Z] [NOTICE] recover called [2017-07-17T06:44:41.268Z] [NOTICE] Ansible Service Broker Started [2017-07-17T06:44:41.268Z] [NOTICE] Listening on http://0.0.0.0:1338 [2017-07-17T06:46:39.239Z] [INFO] ASYNC deprovision in progress [2017-07-17T06:46:39.239Z] [NOTICE] ============================================================ [2017-07-17T06:46:39.239Z] [NOTICE] DEPROVISIONING [2017-07-17T06:46:39.239Z] [NOTICE] ============================================================ [2017-07-17T06:46:39.239Z] [NOTICE] ServiceInstance.Id: 4fbcc051-03af-40c7-86e2-dc12977e6b5d [2017-07-17T06:46:39.239Z] [NOTICE] ServiceInstance.Name: mediawiki-apb [2017-07-17T06:46:39.239Z] [NOTICE] ServiceInstance.Image: openshift3/mediawiki-apb [2017-07-17T06:46:39.239Z] [NOTICE] ServiceInstance.Description: Mediawiki123 apb implementation [2017-07-17T06:46:39.239Z] [NOTICE] ============================================================ 10.129.0.1 - - [17/Jul/2017:06:46:39 +0000] "DELETE /v2/service_instances/7ae48ae2-87aa-4f6e-a293-52d3a5842ee8?accepts_incomplete=true&plan_id=4c10ff42-be89-420a-9bab-27a9bef9aed8&service_id=4fbcc051-03af-40c7-86e2-dc12977e6b5d HTTP/1.1" 200 58 [2017-07-17T06:46:39.243Z] [INFO] Successfully wrote resources to /tmp/asb-resource-files/apb-18ba0ecb-8a7c-420c-8307-1e9d2eb8960c.yaml [2017-07-17T06:46:39.652Z] [ERROR] Something went wrong trying to create resources in cluster [2017-07-17T06:46:39.652Z] [ERROR] Returned error: [2017-07-17T06:46:39.652Z] [ERROR] exit status 1 [2017-07-17T06:46:39.652Z] [ERROR] oc create -f output: [2017-07-17T06:46:39.652Z] [ERROR] Error from server (Forbidden): error when creating "/tmp/asb-resource-files/apb-18ba0ecb-8a7c-420c-8307-1e9d2eb8960c.yaml": User "system:serviceaccount:openshift-ansible-service-broker:asb" cannot create serviceaccounts in project "dma" Error from server (Forbidden): error when creating "/tmp/asb-resource-files/apb-18ba0ecb-8a7c-420c-8307-1e9d2eb8960c.yaml": User "system:serviceaccount:openshift-ansible-service-broker:asb" cannot create rolebindings in project "dma" [2017-07-17T06:46:39.652Z] [INFO] Successfully created apb sandbox: [ apb-18ba0ecb-8a7c-420c-8307-1e9d2eb8960c ] [2017-07-17T06:46:39.652Z] [NOTICE] Creating pod "apb-18ba0ecb-8a7c-420c-8307-1e9d2eb8960c" in the dma namespace [2017-07-17T06:46:39.66Z] [ERROR] Problem running image [2017-07-17T06:46:39.66Z] [INFO] Destroying APB sandbox... [2017-07-17T06:46:39.881Z] [ERROR] Something went wrong trying to destroy the serviceaccount! [2017-07-17T06:46:39.881Z] [ERROR] exit status 1 [2017-07-17T06:46:39.881Z] [ERROR] oc delete output: [2017-07-17T06:46:39.881Z] [ERROR] Error from server (Forbidden): User "system:serviceaccount:openshift-ansible-service-broker:asb" cannot delete serviceaccounts in project "dma" [2017-07-17T06:46:39.881Z] [ERROR] error from deprovision - &errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ListMeta:v1.ListMeta{SelfLink:"", ResourceVersion:""}, Status:"Failure", Message:"User \"system:serviceaccount:openshift-ansible-service-broker:asb\" cannot create pods in project \"dma\"", Reason:"Forbidden", Details:(*v1.StatusDetails)(0xc4201373b0), Code:403}} [2017-07-17T06:46:39.881Z] [ERROR] broker::Deprovision error occurred. [2017-07-17T06:46:39.881Z] [ERROR] User "system:serviceaccount:openshift-ansible-service-broker:asb" cannot create pods in project "dma"
Verify on openshift v3.6.170 and asbd 0.9.10, when project doesn't exist, it will prevent provision, when project exist, it can provision successfully, works as expect. [root@host-8-175-47 dma]# oc rsh asb-3169753244-k3c6b sh-4.2$ asbd --version 0.9.10 sh-4.2$ exit exit [root@host-8-175-47 dma]# oc logs asb-3169753244-k3c6b Using config file mounted to /etc/ansible-service-broker/config.yaml ============================================================ == Starting Ansible Service Broker... == ============================================================ [2017-07-26T05:35:05.665Z] [INFO] == ETCD CX == [2017-07-26T05:35:05.665Z] [INFO] EtcdHost: etcd [2017-07-26T05:35:05.665Z] [INFO] EtcdPort: 2379 [2017-07-26T05:35:05.666Z] [INFO] Endpoints: [http://etcd:2379] [2017-07-26T05:35:05.672Z] [INFO] Etcd Version [Server: 3.2.4, Cluster: 3.2.0] [2017-07-26T05:35:05.673Z] [INFO] Cluster Client Created [2017-07-26T05:35:05.703Z] [INFO] Kubernetes version: v1.6.1+5115d708d7 [2017-07-26T05:35:05.703Z] [INFO] == REGISTRY CX == [2017-07-26T05:35:05.703Z] [INFO] Name: rhcc [2017-07-26T05:35:05.703Z] [INFO] Url: registry.access.stage.redhat.com [2017-07-26T05:35:05.704Z] [INFO] Listening for provision messages [2017-07-26T05:35:06.021Z] [INFO] Initiating Recovery Process [2017-07-26T05:35:06.024Z] [INFO] Recovery complete [2017-07-26T05:35:06.024Z] [NOTICE] recover called [2017-07-26T05:35:06.024Z] [INFO] Broker configured to bootstrap on startup [2017-07-26T05:35:06.024Z] [INFO] Attempting bootstrap... [2017-07-26T05:35:06.024Z] [INFO] AnsibleBroker::Bootstrap [2017-07-26T05:35:07.671Z] [INFO] Didn't find encoded Spec label. Assuming image is not APB and skipping. [2017-07-26T05:35:07.982Z] [NOTICE] Broker successfully bootstrapped on startup [2017-07-26T05:35:07.982Z] [NOTICE] Ansible Service Broker Started [2017-07-26T05:35:07.982Z] [NOTICE] Listening on http://0.0.0.0:1338 [2017-07-26T05:36:42.109Z] [INFO] ASYNC provisioning in progress [2017-07-26T05:36:42.109Z] [NOTICE] ============================================================ [2017-07-26T05:36:42.109Z] [NOTICE] PROVISIONING [2017-07-26T05:36:42.109Z] [NOTICE] ============================================================ [2017-07-26T05:36:42.109Z] [NOTICE] Spec.Id: 4fbcc051-03af-40c7-86e2-dc12977e6b5d [2017-07-26T05:36:42.109Z] [NOTICE] Spec.Name: mediawiki-apb [2017-07-26T05:36:42.11Z] [NOTICE] Spec.Image: openshift3/mediawiki-apb [2017-07-26T05:36:42.11Z] [NOTICE] Spec.Description: Mediawiki123 apb implementation [2017-07-26T05:36:42.11Z] [NOTICE] ============================================================ [2017-07-26T05:36:42.11Z] [INFO] Checking if project dma exists... 10.128.0.1 - - [26/Jul/2017:05:36:42 +0000] "PUT /v2/service_instances/2cd33d57-491a-42d0-9bf9-979b5e136f66?accepts_incomplete=true HTTP/1.1" 202 58 [2017-07-26T05:36:42.409Z] [ERROR] Project dma does NOT exist! Cannot provision requested mediawiki-apb [2017-07-26T05:36:42.409Z] [ERROR] broker::Provision error occurred. [2017-07-26T05:36:42.409Z] [ERROR] Project dma does not exist [2017-07-26T05:36:42.409Z] [ERROR] Attempting to destroy APB sandbox if it has been created [2017-07-26T05:36:42.409Z] [INFO] Requested destruction of APB sandbox with empty handle, skipping. [2017-07-26T05:37:22.982Z] [INFO] ASYNC provisioning in progress [2017-07-26T05:37:22.982Z] [NOTICE] ============================================================ [2017-07-26T05:37:22.982Z] [NOTICE] PROVISIONING [2017-07-26T05:37:22.982Z] [NOTICE] ============================================================ [2017-07-26T05:37:22.982Z] [NOTICE] Spec.Id: 4fbcc051-03af-40c7-86e2-dc12977e6b5d [2017-07-26T05:37:22.982Z] [NOTICE] Spec.Name: mediawiki-apb [2017-07-26T05:37:22.982Z] [NOTICE] Spec.Image: openshift3/mediawiki-apb [2017-07-26T05:37:22.982Z] [NOTICE] Spec.Description: Mediawiki123 apb implementation [2017-07-26T05:37:22.982Z] [NOTICE] ============================================================ [2017-07-26T05:37:22.982Z] [INFO] Checking if project dma exists... 10.128.0.1 - - [26/Jul/2017:05:37:22 +0000] "PUT /v2/service_instances/1ceebfe5-ee94-4d14-8b38-c214dd3e12fc?accepts_incomplete=true HTTP/1.1" 202 58 [2017-07-26T05:37:23.194Z] [INFO] Successfully wrote resources to /tmp/asb-resource-files/apb-aacf11b1-da99-4d5f-94ea-5d114354822e.yaml [2017-07-26T05:37:23.514Z] [INFO] Successfully created apb sandbox: [ apb-aacf11b1-da99-4d5f-94ea-5d114354822e ] [2017-07-26T05:37:23.514Z] [NOTICE] Creating pod "apb-aacf11b1-da99-4d5f-94ea-5d114354822e" in the dma namespace [2017-07-26T05:37:29.744Z] [INFO] Container not up yet, retrying 1 of 150 on pod apb-aacf11b1-da99-4d5f-94ea-5d114354822e [2017-07-26T05:38:03.221Z] [INFO] Destroying APB sandbox...