Bug 1469485

Summary: Need to update service account Ansible Service Broker is using for proper permissions
Product: OpenShift Container Platform Reporter: DeShuai Ma <dma>
Component: Service BrokerAssignee: Fabian von Feilitzsch <fabian>
Status: CLOSED CURRENTRELEASE QA Contact: DeShuai Ma <dma>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.6.0CC: aos-bugs, jmatthew, jokerman, mmccomas, rhallise
Target Milestone: ---   
Target Release: 3.7.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-11-10 20:54:35 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description DeShuai Ma 2017-07-11 11:18:49 UTC
Description of problem:
When provision mediawiki apb, falied with "Something went wrong creating project mediawiki-project!"

Version-Release number of the following components:
openshift v3.6.136
kubernetes v1.6.1+5115d708d7
etcd 3.2.1
openshift-ansible-3.6.140-1.git.0.4a02427.el7.noarch.rpm

How reproducible:
Always

Steps to Reproduce:
1.  When provision mediawiki
[root@ip-172-18-6-17 asb]# cat provison.sh 
#!/bin/bash -e

instanceUUID="5d85082a-c8a9-49e3-82a1-3b6ebe6be644"
planUUID="4c10ff42-be89-420a-9bab-27a9bef9aed8"
serviceUUID="4fbcc051-03af-40c7-86e2-dc12977e6b5d"

req="{
  \"plan_id\": \"$planUUID\",
  \"service_id\": \"$serviceUUID\",
  \"context\": {
    \"platform\": \"kubernetes\",
    \"namespace\": \"mediawiki-project\"
  },
  \"parameters\": {
    \"mediawiki_db_schema\": \"mediawiki\",
    \"mediawiki_site_name\": \"mediawiki\",
    \"mediawiki_site_lang\": \"en\",
    \"mediawiki_admin_user\": \"admin\",
    \"mediawiki_admin_pass\":\"redhat\"
  }
}"

curl \
  -X PUT \
  -H 'X-Broker-API-Version: 2.9' \
  -H 'Content-Type: application/json' \
  -d "$req" \
  -v \
  "http://172.30.112.253:1338/v2/service_instances/$instanceUUID?accepts_incomplete=true"

2. error logs from asb pod:
   [2017-07-11T10:57:20.833Z] [INFO] ASYNC provisioning in progress
    [2017-07-11T10:57:20.833Z] [NOTICE] ============================================================
    [2017-07-11T10:57:20.833Z] [NOTICE]                        PROVISIONING                        
    [2017-07-11T10:57:20.833Z] [NOTICE] ============================================================
    [2017-07-11T10:57:20.833Z] [NOTICE] Spec.ID: 4fbcc051-03af-40c7-86e2-dc12977e6b5d
    [2017-07-11T10:57:20.833Z] [NOTICE] Spec.Name: mediawiki-apb
    [2017-07-11T10:57:20.833Z] [NOTICE] Spec.Image: openshift3/mediawiki-apb
    [2017-07-11T10:57:20.833Z] [NOTICE] Spec.Description: Mediawiki123 apb implementation
    [2017-07-11T10:57:20.836Z] [NOTICE] ============================================================
    [2017-07-11T10:57:20.836Z] [INFO] Checking if project mediawiki-project exists...
    10.129.0.1 - - [11/Jul/2017:10:57:20 +0000] "PUT /v2/service_instances/bdeb07ca-2da8-4742-9312-561c6ec86ae0?accepts_incomplete=true HTTP/1.1" 202 58
    [2017-07-11T10:57:21.229Z] [INFO] Project mediawiki-project does NOT exist, creating project...
    [2017-07-11T10:57:21.579Z] [ERROR] Something went wrong creating project mediawiki-project!
    [2017-07-11T10:57:21.579Z] [ERROR] exit status 1
    [2017-07-11T10:57:21.579Z] [ERROR] broker::Provision error occurred.
    [2017-07-11T10:57:21.579Z] [ERROR] exit status 1
    [2017-07-11T10:57:21.579Z] [ERROR] Attempting to destroy APB sandbox if it has been created
    [2017-07-11T10:57:21.579Z] [INFO] Requested destruction of APB sandbox with empty handle, skipping.

Actual results:

Expected results:

Additional info:

Comment 1 John Matthews 2017-07-11 12:56:42 UTC
DeShuai,

The project needs to exist prior to invoking provision.  The APB is not expected to create the project at this point.  (We can consider allowing project creation post 3.6.0, for 3.6.0 we decided project would exist prior to APB)

Please create the project and then re-test

Comment 2 Ryan Hallisey 2017-07-11 13:59:28 UTC
The error seems to be from `oc new-project mediawiki-project`.  Can you run that by hand and paste the output?

Comment 4 DeShuai Ma 2017-07-11 15:09:47 UTC
Eventhrough there is already exist the project it still failed.
After further debug, The issue here is the service account has no permission get/create project. we don't need create the project before provision. If no project, it will be created. After add clusteRrole 'cluster-admin'(actually we don't need so high role) the the sa 'asb', it can provision.

[root@ip-172-18-6-17 asb]# oadm policy add-cluster-role-to-user cluster-admin system:serviceaccount:openshift-ansible-service-broker:asb
cluster role "cluster-admin" added: "system:serviceaccount:openshift-ansible-service-broker:asb"

[2017-07-11T15:01:36.086Z] [INFO] ASYNC provisioning in progress
[2017-07-11T15:01:36.086Z] [NOTICE] ============================================================
[2017-07-11T15:01:36.086Z] [NOTICE]                        PROVISIONING                         
[2017-07-11T15:01:36.086Z] [NOTICE] ============================================================
[2017-07-11T15:01:36.086Z] [NOTICE] Spec.Id: 4fbcc051-03af-40c7-86e2-dc12977e6b5d
[2017-07-11T15:01:36.086Z] [NOTICE] Spec.Name: mediawiki-apb
[2017-07-11T15:01:36.086Z] [NOTICE] Spec.Image: openshift3/mediawiki-apb
[2017-07-11T15:01:36.086Z] [NOTICE] Spec.Description: Mediawiki123 apb implementation
[2017-07-11T15:01:36.086Z] [NOTICE] ============================================================
[2017-07-11T15:01:36.086Z] [INFO] Checking if project mediawiki-project exists...
10.129.0.1 - - [11/Jul/2017:15:01:36 +0000] "PUT /v2/service_instances/5e1745fa-3025-44e8-8998-acce0557caa8?accepts_incomplete=true HTTP/1.1" 202 58
[2017-07-11T15:01:36.542Z] [INFO] Project mediawiki-project does NOT exist, creating project...
[2017-07-11T15:01:37.586Z] [INFO] Successfully created project mediawiki-project
[2017-07-11T15:01:37.586Z] [DEBUG] oc new-project output:
[2017-07-11T15:01:37.586Z] [DEBUG] Now using project "mediawiki-project" on server "https://172.30.0.1:443".
You can add applications to this project with the 'new-app' command. For example, try:
    oc new-app centos/ruby-22-centos7~https://github.com/openshift/ruby-ex.git
to build a new example application in Ruby.

[2017-07-11T15:01:37.587Z] [DEBUG] clusterConfig:
[2017-07-11T15:01:37.587Z] [DEBUG] name:[ mediawiki-apb ]
[2017-07-11T15:01:37.587Z] [DEBUG] image:[ openshift3/mediawiki-apb ]
[2017-07-11T15:01:37.587Z] [DEBUG] action:[ provision ]
[2017-07-11T15:01:37.587Z] [DEBUG] Creating resource file dir: /tmp/asb-resource-files
[2017-07-11T15:01:37.587Z] [DEBUG] Creating resource file /tmp/asb-resource-files/apb-0a3664f9-0e14-45ff-b2e5-6aa18f164aff.yaml
[2017-07-11T15:01:37.588Z] [INFO] Successfully wrote resources to /tmp/asb-resource-files/apb-0a3664f9-0e14-45ff-b2e5-6aa18f164aff.yaml
[2017-07-11T15:01:37.588Z] [DEBUG] Creating resources from file at path: /tmp/asb-resource-files/apb-0a3664f9-0e14-45ff-b2e5-6aa18f164aff.yaml
[2017-07-11T15:01:38.18Z] [DEBUG] Successfully created resources, oc create -f output:
[2017-07-11T15:01:38.18Z] [DEBUG] 
serviceaccount "apb-0a3664f9-0e14-45ff-b2e5-6aa18f164aff" created
rolebinding "apb-0a3664f9-0e14-45ff-b2e5-6aa18f164aff" created

[2017-07-11T15:01:38.18Z] [INFO] Successfully created apb sandbox: [ apb-0a3664f9-0e14-45ff-b2e5-6aa18f164aff ]
[2017-07-11T15:01:38.18Z] [NOTICE] Creating pod "apb-0a3664f9-0e14-45ff-b2e5-6aa18f164aff" in the mediawiki-project namespace
[2017-07-11T15:01:39.379Z] [DEBUG] Calling monitorOutput on apb-0a3664f9-0e14-45ff-b2e5-6aa18f164aff
[2017-07-11T15:01:39.903Z] [DEBUG] oc log output: Error from server (BadRequest): container "apb" in pod "apb-0a3664f9-0e14-45ff-b2e5-6aa18f164aff" is waiting to start: ContainerCreating

[2017-07-11T15:01:45.904Z] [INFO] Container not up yet, retrying 1 of 150 on pod apb-0a3664f9-0e14-45ff-b2e5-6aa18f164aff
[2017-07-11T15:01:46.392Z] [DEBUG] oc log output: 
Error from server (BadRequest): container "apb" in pod "apb-0a3664f9-0e14-45ff-b2e5-6aa18f164aff" is waiting to start: image can't be pulled

[2017-07-11T15:01:46.392Z] [NOTICE] WARNING: Unexpected output from apb pod
[2017-07-11T15:01:46.392Z] [NOTICE] Will keep retrying, but it's possible something has gone wrong.
[2017-07-11T15:01:46.392Z] [NOTICE] image can't be pulled
[2017-07-11T15:01:52.392Z] [INFO] Container not up yet, retrying 2 of 150 on pod apb-0a3664f9-0e14-45ff-b2e5-6aa18f164aff
[2017-07-11T15:01:52.828Z] [DEBUG] oc log output: 
Error from server (BadRequest): container "apb" in pod "apb-0a3664f9-0e14-45ff-b2e5-6aa18f164aff" is waiting to start: image can't be pulled

[2017-07-11T15:01:52.829Z] [NOTICE] WARNING: Unexpected output from apb pod
[2017-07-11T15:01:52.829Z] [NOTICE] Will keep retrying, but it's possible something has gone wrong.
[2017-07-11T15:01:52.829Z] [NOTICE] image can't be pulled

Comment 5 DeShuai Ma 2017-07-11 15:12:27 UTC
sorry mistake set the assign back. correct it.

Comment 6 Ryan Hallisey 2017-07-11 15:34:50 UTC
Looks like we need to add something like this to the installer: oadm policy add-cluster-role-to-user <role> system:serviceaccount:openshift-ansible-service-broker:asb

Looking at some of the docs, I think role as basic-user makes sense.

Comment 7 John Matthews 2017-07-11 17:12:34 UTC
Agreed, we need this change in the installer for 3.6.0

Note the broker will need the ability to create service accounts in other projects than it's own.  

We have been using 'cluster-admin' for our testing.

I don't think basic-user will be sufficient.

Comment 8 Ryan Hallisey 2017-07-11 17:39:57 UTC
tracking here: https://github.com/openshift/openshift-ansible/pull/4736

Comment 9 DeShuai Ma 2017-07-17 06:52:41 UTC
system:serviceaccount:openshift-ansible-service-broker:asb also need permission to create/delete sa/pod in other project

[root@host-8-175-186 dma]# oc logs asb-1222154024-1wkbz 
Using config file mounted to /etc/ansible-service-broker/config.yaml
============================================================
==           Starting Ansible Service Broker...           ==
============================================================
[2017-07-17T06:44:40.889Z] [INFO] == ETCD CX ==
[2017-07-17T06:44:40.889Z] [INFO] EtcdHost: etcd
[2017-07-17T06:44:40.889Z] [INFO] EtcdPort: 2379
[2017-07-17T06:44:40.889Z] [INFO] Endpoints: [http://etcd:2379]
[2017-07-17T06:44:40.898Z] [INFO] Etcd Version [Server: 3.2.3, Cluster: 3.2.0]
[2017-07-17T06:44:40.899Z] [INFO] Cluster Client Created
[2017-07-17T06:44:40.941Z] [INFO] Kubernetes version: v1.6.1+5115d708d7
[2017-07-17T06:44:40.941Z] [INFO] == REGISTRY CX == 
[2017-07-17T06:44:40.941Z] [INFO] Name: rhcc
[2017-07-17T06:44:40.941Z] [INFO] Url: registry.access.stage.redhat.com
[2017-07-17T06:44:40.942Z] [INFO] Listening for provision messages
[2017-07-17T06:44:41.265Z] [INFO] Initiating Recovery Process
[2017-07-17T06:44:41.268Z] [INFO] Recovery complete
[2017-07-17T06:44:41.268Z] [NOTICE] recover called
[2017-07-17T06:44:41.268Z] [NOTICE] Ansible Service Broker Started
[2017-07-17T06:44:41.268Z] [NOTICE] Listening on http://0.0.0.0:1338
[2017-07-17T06:46:39.239Z] [INFO] ASYNC deprovision in progress
[2017-07-17T06:46:39.239Z] [NOTICE] ============================================================
[2017-07-17T06:46:39.239Z] [NOTICE]                       DEPROVISIONING                        
[2017-07-17T06:46:39.239Z] [NOTICE] ============================================================
[2017-07-17T06:46:39.239Z] [NOTICE] ServiceInstance.Id: 4fbcc051-03af-40c7-86e2-dc12977e6b5d
[2017-07-17T06:46:39.239Z] [NOTICE] ServiceInstance.Name: mediawiki-apb
[2017-07-17T06:46:39.239Z] [NOTICE] ServiceInstance.Image: openshift3/mediawiki-apb
[2017-07-17T06:46:39.239Z] [NOTICE] ServiceInstance.Description: Mediawiki123 apb implementation
[2017-07-17T06:46:39.239Z] [NOTICE] ============================================================
10.129.0.1 - - [17/Jul/2017:06:46:39 +0000] "DELETE /v2/service_instances/7ae48ae2-87aa-4f6e-a293-52d3a5842ee8?accepts_incomplete=true&plan_id=4c10ff42-be89-420a-9bab-27a9bef9aed8&service_id=4fbcc051-03af-40c7-86e2-dc12977e6b5d HTTP/1.1" 200 58
[2017-07-17T06:46:39.243Z] [INFO] Successfully wrote resources to /tmp/asb-resource-files/apb-18ba0ecb-8a7c-420c-8307-1e9d2eb8960c.yaml
[2017-07-17T06:46:39.652Z] [ERROR] Something went wrong trying to create resources in cluster
[2017-07-17T06:46:39.652Z] [ERROR] Returned error:
[2017-07-17T06:46:39.652Z] [ERROR] exit status 1
[2017-07-17T06:46:39.652Z] [ERROR] oc create -f output:
[2017-07-17T06:46:39.652Z] [ERROR] Error from server (Forbidden): error when creating "/tmp/asb-resource-files/apb-18ba0ecb-8a7c-420c-8307-1e9d2eb8960c.yaml": User "system:serviceaccount:openshift-ansible-service-broker:asb" cannot create serviceaccounts in project "dma"
Error from server (Forbidden): error when creating "/tmp/asb-resource-files/apb-18ba0ecb-8a7c-420c-8307-1e9d2eb8960c.yaml": User "system:serviceaccount:openshift-ansible-service-broker:asb" cannot create rolebindings in project "dma"

[2017-07-17T06:46:39.652Z] [INFO] Successfully created apb sandbox: [ apb-18ba0ecb-8a7c-420c-8307-1e9d2eb8960c ]
[2017-07-17T06:46:39.652Z] [NOTICE] Creating pod "apb-18ba0ecb-8a7c-420c-8307-1e9d2eb8960c" in the dma namespace
[2017-07-17T06:46:39.66Z] [ERROR] Problem running image
[2017-07-17T06:46:39.66Z] [INFO] Destroying APB sandbox...
[2017-07-17T06:46:39.881Z] [ERROR] Something went wrong trying to destroy the serviceaccount!
[2017-07-17T06:46:39.881Z] [ERROR] exit status 1
[2017-07-17T06:46:39.881Z] [ERROR] oc delete output:
[2017-07-17T06:46:39.881Z] [ERROR] Error from server (Forbidden): User "system:serviceaccount:openshift-ansible-service-broker:asb" cannot delete serviceaccounts in project "dma"

[2017-07-17T06:46:39.881Z] [ERROR] error from deprovision - &errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ListMeta:v1.ListMeta{SelfLink:"", ResourceVersion:""}, Status:"Failure", Message:"User \"system:serviceaccount:openshift-ansible-service-broker:asb\" cannot create pods in project \"dma\"", Reason:"Forbidden", Details:(*v1.StatusDetails)(0xc4201373b0), Code:403}}
[2017-07-17T06:46:39.881Z] [ERROR] broker::Deprovision error occurred.
[2017-07-17T06:46:39.881Z] [ERROR] User "system:serviceaccount:openshift-ansible-service-broker:asb" cannot create pods in project "dma"

Comment 11 DeShuai Ma 2017-07-26 05:42:56 UTC
Verify on openshift v3.6.170 and asbd 0.9.10, when project doesn't exist, it will prevent provision, when project exist, it can provision successfully, works as expect.

[root@host-8-175-47 dma]# oc rsh asb-3169753244-k3c6b
sh-4.2$ asbd --version
0.9.10
sh-4.2$ exit
exit
[root@host-8-175-47 dma]# oc logs asb-3169753244-k3c6b 
Using config file mounted to /etc/ansible-service-broker/config.yaml
============================================================
==           Starting Ansible Service Broker...           ==
============================================================
[2017-07-26T05:35:05.665Z] [INFO] == ETCD CX ==
[2017-07-26T05:35:05.665Z] [INFO] EtcdHost: etcd
[2017-07-26T05:35:05.665Z] [INFO] EtcdPort: 2379
[2017-07-26T05:35:05.666Z] [INFO] Endpoints: [http://etcd:2379]
[2017-07-26T05:35:05.672Z] [INFO] Etcd Version [Server: 3.2.4, Cluster: 3.2.0]
[2017-07-26T05:35:05.673Z] [INFO] Cluster Client Created
[2017-07-26T05:35:05.703Z] [INFO] Kubernetes version: v1.6.1+5115d708d7
[2017-07-26T05:35:05.703Z] [INFO] == REGISTRY CX == 
[2017-07-26T05:35:05.703Z] [INFO] Name: rhcc
[2017-07-26T05:35:05.703Z] [INFO] Url: registry.access.stage.redhat.com
[2017-07-26T05:35:05.704Z] [INFO] Listening for provision messages
[2017-07-26T05:35:06.021Z] [INFO] Initiating Recovery Process
[2017-07-26T05:35:06.024Z] [INFO] Recovery complete
[2017-07-26T05:35:06.024Z] [NOTICE] recover called
[2017-07-26T05:35:06.024Z] [INFO] Broker configured to bootstrap on startup
[2017-07-26T05:35:06.024Z] [INFO] Attempting bootstrap...
[2017-07-26T05:35:06.024Z] [INFO] AnsibleBroker::Bootstrap
[2017-07-26T05:35:07.671Z] [INFO] Didn't find encoded Spec label. Assuming image is not APB and skipping.
[2017-07-26T05:35:07.982Z] [NOTICE] Broker successfully bootstrapped on startup
[2017-07-26T05:35:07.982Z] [NOTICE] Ansible Service Broker Started
[2017-07-26T05:35:07.982Z] [NOTICE] Listening on http://0.0.0.0:1338
[2017-07-26T05:36:42.109Z] [INFO] ASYNC provisioning in progress
[2017-07-26T05:36:42.109Z] [NOTICE] ============================================================
[2017-07-26T05:36:42.109Z] [NOTICE]                        PROVISIONING                         
[2017-07-26T05:36:42.109Z] [NOTICE] ============================================================
[2017-07-26T05:36:42.109Z] [NOTICE] Spec.Id: 4fbcc051-03af-40c7-86e2-dc12977e6b5d
[2017-07-26T05:36:42.109Z] [NOTICE] Spec.Name: mediawiki-apb
[2017-07-26T05:36:42.11Z] [NOTICE] Spec.Image: openshift3/mediawiki-apb
[2017-07-26T05:36:42.11Z] [NOTICE] Spec.Description: Mediawiki123 apb implementation
[2017-07-26T05:36:42.11Z] [NOTICE] ============================================================
[2017-07-26T05:36:42.11Z] [INFO] Checking if project dma exists...
10.128.0.1 - - [26/Jul/2017:05:36:42 +0000] "PUT /v2/service_instances/2cd33d57-491a-42d0-9bf9-979b5e136f66?accepts_incomplete=true HTTP/1.1" 202 58
[2017-07-26T05:36:42.409Z] [ERROR] Project dma does NOT exist! Cannot provision requested mediawiki-apb
[2017-07-26T05:36:42.409Z] [ERROR] broker::Provision error occurred.
[2017-07-26T05:36:42.409Z] [ERROR] Project dma does not exist
[2017-07-26T05:36:42.409Z] [ERROR] Attempting to destroy APB sandbox if it has been created
[2017-07-26T05:36:42.409Z] [INFO] Requested destruction of APB sandbox with empty handle, skipping.
[2017-07-26T05:37:22.982Z] [INFO] ASYNC provisioning in progress
[2017-07-26T05:37:22.982Z] [NOTICE] ============================================================
[2017-07-26T05:37:22.982Z] [NOTICE]                        PROVISIONING                         
[2017-07-26T05:37:22.982Z] [NOTICE] ============================================================
[2017-07-26T05:37:22.982Z] [NOTICE] Spec.Id: 4fbcc051-03af-40c7-86e2-dc12977e6b5d
[2017-07-26T05:37:22.982Z] [NOTICE] Spec.Name: mediawiki-apb
[2017-07-26T05:37:22.982Z] [NOTICE] Spec.Image: openshift3/mediawiki-apb
[2017-07-26T05:37:22.982Z] [NOTICE] Spec.Description: Mediawiki123 apb implementation
[2017-07-26T05:37:22.982Z] [NOTICE] ============================================================
[2017-07-26T05:37:22.982Z] [INFO] Checking if project dma exists...
10.128.0.1 - - [26/Jul/2017:05:37:22 +0000] "PUT /v2/service_instances/1ceebfe5-ee94-4d14-8b38-c214dd3e12fc?accepts_incomplete=true HTTP/1.1" 202 58
[2017-07-26T05:37:23.194Z] [INFO] Successfully wrote resources to /tmp/asb-resource-files/apb-aacf11b1-da99-4d5f-94ea-5d114354822e.yaml
[2017-07-26T05:37:23.514Z] [INFO] Successfully created apb sandbox: [ apb-aacf11b1-da99-4d5f-94ea-5d114354822e ]
[2017-07-26T05:37:23.514Z] [NOTICE] Creating pod "apb-aacf11b1-da99-4d5f-94ea-5d114354822e" in the dma namespace
[2017-07-26T05:37:29.744Z] [INFO] Container not up yet, retrying 1 of 150 on pod apb-aacf11b1-da99-4d5f-94ea-5d114354822e
[2017-07-26T05:38:03.221Z] [INFO] Destroying APB sandbox...