Bug 1469485 - Need to update service account Ansible Service Broker is using for proper permissions
Summary: Need to update service account Ansible Service Broker is using for proper per...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Service Broker
Version: 3.6.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 3.7.0
Assignee: Fabian von Feilitzsch
QA Contact: DeShuai Ma
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-07-11 11:18 UTC by DeShuai Ma
Modified: 2017-11-28 08:28 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-11-10 20:54:35 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2017:3188 normal SHIPPED_LIVE Moderate: Red Hat OpenShift Container Platform 3.7 security, bug, and enhancement update 2017-11-29 02:34:54 UTC

Description DeShuai Ma 2017-07-11 11:18:49 UTC
Description of problem:
When provision mediawiki apb, falied with "Something went wrong creating project mediawiki-project!"

Version-Release number of the following components:
openshift v3.6.136
kubernetes v1.6.1+5115d708d7
etcd 3.2.1
openshift-ansible-3.6.140-1.git.0.4a02427.el7.noarch.rpm

How reproducible:
Always

Steps to Reproduce:
1.  When provision mediawiki
[root@ip-172-18-6-17 asb]# cat provison.sh 
#!/bin/bash -e

instanceUUID="5d85082a-c8a9-49e3-82a1-3b6ebe6be644"
planUUID="4c10ff42-be89-420a-9bab-27a9bef9aed8"
serviceUUID="4fbcc051-03af-40c7-86e2-dc12977e6b5d"

req="{
  \"plan_id\": \"$planUUID\",
  \"service_id\": \"$serviceUUID\",
  \"context\": {
    \"platform\": \"kubernetes\",
    \"namespace\": \"mediawiki-project\"
  },
  \"parameters\": {
    \"mediawiki_db_schema\": \"mediawiki\",
    \"mediawiki_site_name\": \"mediawiki\",
    \"mediawiki_site_lang\": \"en\",
    \"mediawiki_admin_user\": \"admin\",
    \"mediawiki_admin_pass\":\"redhat\"
  }
}"

curl \
  -X PUT \
  -H 'X-Broker-API-Version: 2.9' \
  -H 'Content-Type: application/json' \
  -d "$req" \
  -v \
  "http://172.30.112.253:1338/v2/service_instances/$instanceUUID?accepts_incomplete=true"

2. error logs from asb pod:
   [2017-07-11T10:57:20.833Z] [INFO] ASYNC provisioning in progress
    [2017-07-11T10:57:20.833Z] [NOTICE] ============================================================
    [2017-07-11T10:57:20.833Z] [NOTICE]                        PROVISIONING                        
    [2017-07-11T10:57:20.833Z] [NOTICE] ============================================================
    [2017-07-11T10:57:20.833Z] [NOTICE] Spec.ID: 4fbcc051-03af-40c7-86e2-dc12977e6b5d
    [2017-07-11T10:57:20.833Z] [NOTICE] Spec.Name: mediawiki-apb
    [2017-07-11T10:57:20.833Z] [NOTICE] Spec.Image: openshift3/mediawiki-apb
    [2017-07-11T10:57:20.833Z] [NOTICE] Spec.Description: Mediawiki123 apb implementation
    [2017-07-11T10:57:20.836Z] [NOTICE] ============================================================
    [2017-07-11T10:57:20.836Z] [INFO] Checking if project mediawiki-project exists...
    10.129.0.1 - - [11/Jul/2017:10:57:20 +0000] "PUT /v2/service_instances/bdeb07ca-2da8-4742-9312-561c6ec86ae0?accepts_incomplete=true HTTP/1.1" 202 58
    [2017-07-11T10:57:21.229Z] [INFO] Project mediawiki-project does NOT exist, creating project...
    [2017-07-11T10:57:21.579Z] [ERROR] Something went wrong creating project mediawiki-project!
    [2017-07-11T10:57:21.579Z] [ERROR] exit status 1
    [2017-07-11T10:57:21.579Z] [ERROR] broker::Provision error occurred.
    [2017-07-11T10:57:21.579Z] [ERROR] exit status 1
    [2017-07-11T10:57:21.579Z] [ERROR] Attempting to destroy APB sandbox if it has been created
    [2017-07-11T10:57:21.579Z] [INFO] Requested destruction of APB sandbox with empty handle, skipping.

Actual results:

Expected results:

Additional info:

Comment 1 John Matthews 2017-07-11 12:56:42 UTC
DeShuai,

The project needs to exist prior to invoking provision.  The APB is not expected to create the project at this point.  (We can consider allowing project creation post 3.6.0, for 3.6.0 we decided project would exist prior to APB)

Please create the project and then re-test

Comment 2 Ryan Hallisey 2017-07-11 13:59:28 UTC
The error seems to be from `oc new-project mediawiki-project`.  Can you run that by hand and paste the output?

Comment 4 DeShuai Ma 2017-07-11 15:09:47 UTC
Eventhrough there is already exist the project it still failed.
After further debug, The issue here is the service account has no permission get/create project. we don't need create the project before provision. If no project, it will be created. After add clusteRrole 'cluster-admin'(actually we don't need so high role) the the sa 'asb', it can provision.

[root@ip-172-18-6-17 asb]# oadm policy add-cluster-role-to-user cluster-admin system:serviceaccount:openshift-ansible-service-broker:asb
cluster role "cluster-admin" added: "system:serviceaccount:openshift-ansible-service-broker:asb"

[2017-07-11T15:01:36.086Z] [INFO] ASYNC provisioning in progress
[2017-07-11T15:01:36.086Z] [NOTICE] ============================================================
[2017-07-11T15:01:36.086Z] [NOTICE]                        PROVISIONING                         
[2017-07-11T15:01:36.086Z] [NOTICE] ============================================================
[2017-07-11T15:01:36.086Z] [NOTICE] Spec.Id: 4fbcc051-03af-40c7-86e2-dc12977e6b5d
[2017-07-11T15:01:36.086Z] [NOTICE] Spec.Name: mediawiki-apb
[2017-07-11T15:01:36.086Z] [NOTICE] Spec.Image: openshift3/mediawiki-apb
[2017-07-11T15:01:36.086Z] [NOTICE] Spec.Description: Mediawiki123 apb implementation
[2017-07-11T15:01:36.086Z] [NOTICE] ============================================================
[2017-07-11T15:01:36.086Z] [INFO] Checking if project mediawiki-project exists...
10.129.0.1 - - [11/Jul/2017:15:01:36 +0000] "PUT /v2/service_instances/5e1745fa-3025-44e8-8998-acce0557caa8?accepts_incomplete=true HTTP/1.1" 202 58
[2017-07-11T15:01:36.542Z] [INFO] Project mediawiki-project does NOT exist, creating project...
[2017-07-11T15:01:37.586Z] [INFO] Successfully created project mediawiki-project
[2017-07-11T15:01:37.586Z] [DEBUG] oc new-project output:
[2017-07-11T15:01:37.586Z] [DEBUG] Now using project "mediawiki-project" on server "https://172.30.0.1:443".
You can add applications to this project with the 'new-app' command. For example, try:
    oc new-app centos/ruby-22-centos7~https://github.com/openshift/ruby-ex.git
to build a new example application in Ruby.

[2017-07-11T15:01:37.587Z] [DEBUG] clusterConfig:
[2017-07-11T15:01:37.587Z] [DEBUG] name:[ mediawiki-apb ]
[2017-07-11T15:01:37.587Z] [DEBUG] image:[ openshift3/mediawiki-apb ]
[2017-07-11T15:01:37.587Z] [DEBUG] action:[ provision ]
[2017-07-11T15:01:37.587Z] [DEBUG] Creating resource file dir: /tmp/asb-resource-files
[2017-07-11T15:01:37.587Z] [DEBUG] Creating resource file /tmp/asb-resource-files/apb-0a3664f9-0e14-45ff-b2e5-6aa18f164aff.yaml
[2017-07-11T15:01:37.588Z] [INFO] Successfully wrote resources to /tmp/asb-resource-files/apb-0a3664f9-0e14-45ff-b2e5-6aa18f164aff.yaml
[2017-07-11T15:01:37.588Z] [DEBUG] Creating resources from file at path: /tmp/asb-resource-files/apb-0a3664f9-0e14-45ff-b2e5-6aa18f164aff.yaml
[2017-07-11T15:01:38.18Z] [DEBUG] Successfully created resources, oc create -f output:
[2017-07-11T15:01:38.18Z] [DEBUG] 
serviceaccount "apb-0a3664f9-0e14-45ff-b2e5-6aa18f164aff" created
rolebinding "apb-0a3664f9-0e14-45ff-b2e5-6aa18f164aff" created

[2017-07-11T15:01:38.18Z] [INFO] Successfully created apb sandbox: [ apb-0a3664f9-0e14-45ff-b2e5-6aa18f164aff ]
[2017-07-11T15:01:38.18Z] [NOTICE] Creating pod "apb-0a3664f9-0e14-45ff-b2e5-6aa18f164aff" in the mediawiki-project namespace
[2017-07-11T15:01:39.379Z] [DEBUG] Calling monitorOutput on apb-0a3664f9-0e14-45ff-b2e5-6aa18f164aff
[2017-07-11T15:01:39.903Z] [DEBUG] oc log output: Error from server (BadRequest): container "apb" in pod "apb-0a3664f9-0e14-45ff-b2e5-6aa18f164aff" is waiting to start: ContainerCreating

[2017-07-11T15:01:45.904Z] [INFO] Container not up yet, retrying 1 of 150 on pod apb-0a3664f9-0e14-45ff-b2e5-6aa18f164aff
[2017-07-11T15:01:46.392Z] [DEBUG] oc log output: 
Error from server (BadRequest): container "apb" in pod "apb-0a3664f9-0e14-45ff-b2e5-6aa18f164aff" is waiting to start: image can't be pulled

[2017-07-11T15:01:46.392Z] [NOTICE] WARNING: Unexpected output from apb pod
[2017-07-11T15:01:46.392Z] [NOTICE] Will keep retrying, but it's possible something has gone wrong.
[2017-07-11T15:01:46.392Z] [NOTICE] image can't be pulled
[2017-07-11T15:01:52.392Z] [INFO] Container not up yet, retrying 2 of 150 on pod apb-0a3664f9-0e14-45ff-b2e5-6aa18f164aff
[2017-07-11T15:01:52.828Z] [DEBUG] oc log output: 
Error from server (BadRequest): container "apb" in pod "apb-0a3664f9-0e14-45ff-b2e5-6aa18f164aff" is waiting to start: image can't be pulled

[2017-07-11T15:01:52.829Z] [NOTICE] WARNING: Unexpected output from apb pod
[2017-07-11T15:01:52.829Z] [NOTICE] Will keep retrying, but it's possible something has gone wrong.
[2017-07-11T15:01:52.829Z] [NOTICE] image can't be pulled

Comment 5 DeShuai Ma 2017-07-11 15:12:27 UTC
sorry mistake set the assign back. correct it.

Comment 6 Ryan Hallisey 2017-07-11 15:34:50 UTC
Looks like we need to add something like this to the installer: oadm policy add-cluster-role-to-user <role> system:serviceaccount:openshift-ansible-service-broker:asb

Looking at some of the docs, I think role as basic-user makes sense.

Comment 7 John Matthews 2017-07-11 17:12:34 UTC
Agreed, we need this change in the installer for 3.6.0

Note the broker will need the ability to create service accounts in other projects than it's own.  

We have been using 'cluster-admin' for our testing.

I don't think basic-user will be sufficient.

Comment 8 Ryan Hallisey 2017-07-11 17:39:57 UTC
tracking here: https://github.com/openshift/openshift-ansible/pull/4736

Comment 9 DeShuai Ma 2017-07-17 06:52:41 UTC
system:serviceaccount:openshift-ansible-service-broker:asb also need permission to create/delete sa/pod in other project

[root@host-8-175-186 dma]# oc logs asb-1222154024-1wkbz 
Using config file mounted to /etc/ansible-service-broker/config.yaml
============================================================
==           Starting Ansible Service Broker...           ==
============================================================
[2017-07-17T06:44:40.889Z] [INFO] == ETCD CX ==
[2017-07-17T06:44:40.889Z] [INFO] EtcdHost: etcd
[2017-07-17T06:44:40.889Z] [INFO] EtcdPort: 2379
[2017-07-17T06:44:40.889Z] [INFO] Endpoints: [http://etcd:2379]
[2017-07-17T06:44:40.898Z] [INFO] Etcd Version [Server: 3.2.3, Cluster: 3.2.0]
[2017-07-17T06:44:40.899Z] [INFO] Cluster Client Created
[2017-07-17T06:44:40.941Z] [INFO] Kubernetes version: v1.6.1+5115d708d7
[2017-07-17T06:44:40.941Z] [INFO] == REGISTRY CX == 
[2017-07-17T06:44:40.941Z] [INFO] Name: rhcc
[2017-07-17T06:44:40.941Z] [INFO] Url: registry.access.stage.redhat.com
[2017-07-17T06:44:40.942Z] [INFO] Listening for provision messages
[2017-07-17T06:44:41.265Z] [INFO] Initiating Recovery Process
[2017-07-17T06:44:41.268Z] [INFO] Recovery complete
[2017-07-17T06:44:41.268Z] [NOTICE] recover called
[2017-07-17T06:44:41.268Z] [NOTICE] Ansible Service Broker Started
[2017-07-17T06:44:41.268Z] [NOTICE] Listening on http://0.0.0.0:1338
[2017-07-17T06:46:39.239Z] [INFO] ASYNC deprovision in progress
[2017-07-17T06:46:39.239Z] [NOTICE] ============================================================
[2017-07-17T06:46:39.239Z] [NOTICE]                       DEPROVISIONING                        
[2017-07-17T06:46:39.239Z] [NOTICE] ============================================================
[2017-07-17T06:46:39.239Z] [NOTICE] ServiceInstance.Id: 4fbcc051-03af-40c7-86e2-dc12977e6b5d
[2017-07-17T06:46:39.239Z] [NOTICE] ServiceInstance.Name: mediawiki-apb
[2017-07-17T06:46:39.239Z] [NOTICE] ServiceInstance.Image: openshift3/mediawiki-apb
[2017-07-17T06:46:39.239Z] [NOTICE] ServiceInstance.Description: Mediawiki123 apb implementation
[2017-07-17T06:46:39.239Z] [NOTICE] ============================================================
10.129.0.1 - - [17/Jul/2017:06:46:39 +0000] "DELETE /v2/service_instances/7ae48ae2-87aa-4f6e-a293-52d3a5842ee8?accepts_incomplete=true&plan_id=4c10ff42-be89-420a-9bab-27a9bef9aed8&service_id=4fbcc051-03af-40c7-86e2-dc12977e6b5d HTTP/1.1" 200 58
[2017-07-17T06:46:39.243Z] [INFO] Successfully wrote resources to /tmp/asb-resource-files/apb-18ba0ecb-8a7c-420c-8307-1e9d2eb8960c.yaml
[2017-07-17T06:46:39.652Z] [ERROR] Something went wrong trying to create resources in cluster
[2017-07-17T06:46:39.652Z] [ERROR] Returned error:
[2017-07-17T06:46:39.652Z] [ERROR] exit status 1
[2017-07-17T06:46:39.652Z] [ERROR] oc create -f output:
[2017-07-17T06:46:39.652Z] [ERROR] Error from server (Forbidden): error when creating "/tmp/asb-resource-files/apb-18ba0ecb-8a7c-420c-8307-1e9d2eb8960c.yaml": User "system:serviceaccount:openshift-ansible-service-broker:asb" cannot create serviceaccounts in project "dma"
Error from server (Forbidden): error when creating "/tmp/asb-resource-files/apb-18ba0ecb-8a7c-420c-8307-1e9d2eb8960c.yaml": User "system:serviceaccount:openshift-ansible-service-broker:asb" cannot create rolebindings in project "dma"

[2017-07-17T06:46:39.652Z] [INFO] Successfully created apb sandbox: [ apb-18ba0ecb-8a7c-420c-8307-1e9d2eb8960c ]
[2017-07-17T06:46:39.652Z] [NOTICE] Creating pod "apb-18ba0ecb-8a7c-420c-8307-1e9d2eb8960c" in the dma namespace
[2017-07-17T06:46:39.66Z] [ERROR] Problem running image
[2017-07-17T06:46:39.66Z] [INFO] Destroying APB sandbox...
[2017-07-17T06:46:39.881Z] [ERROR] Something went wrong trying to destroy the serviceaccount!
[2017-07-17T06:46:39.881Z] [ERROR] exit status 1
[2017-07-17T06:46:39.881Z] [ERROR] oc delete output:
[2017-07-17T06:46:39.881Z] [ERROR] Error from server (Forbidden): User "system:serviceaccount:openshift-ansible-service-broker:asb" cannot delete serviceaccounts in project "dma"

[2017-07-17T06:46:39.881Z] [ERROR] error from deprovision - &errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ListMeta:v1.ListMeta{SelfLink:"", ResourceVersion:""}, Status:"Failure", Message:"User \"system:serviceaccount:openshift-ansible-service-broker:asb\" cannot create pods in project \"dma\"", Reason:"Forbidden", Details:(*v1.StatusDetails)(0xc4201373b0), Code:403}}
[2017-07-17T06:46:39.881Z] [ERROR] broker::Deprovision error occurred.
[2017-07-17T06:46:39.881Z] [ERROR] User "system:serviceaccount:openshift-ansible-service-broker:asb" cannot create pods in project "dma"

Comment 11 DeShuai Ma 2017-07-26 05:42:56 UTC
Verify on openshift v3.6.170 and asbd 0.9.10, when project doesn't exist, it will prevent provision, when project exist, it can provision successfully, works as expect.

[root@host-8-175-47 dma]# oc rsh asb-3169753244-k3c6b
sh-4.2$ asbd --version
0.9.10
sh-4.2$ exit
exit
[root@host-8-175-47 dma]# oc logs asb-3169753244-k3c6b 
Using config file mounted to /etc/ansible-service-broker/config.yaml
============================================================
==           Starting Ansible Service Broker...           ==
============================================================
[2017-07-26T05:35:05.665Z] [INFO] == ETCD CX ==
[2017-07-26T05:35:05.665Z] [INFO] EtcdHost: etcd
[2017-07-26T05:35:05.665Z] [INFO] EtcdPort: 2379
[2017-07-26T05:35:05.666Z] [INFO] Endpoints: [http://etcd:2379]
[2017-07-26T05:35:05.672Z] [INFO] Etcd Version [Server: 3.2.4, Cluster: 3.2.0]
[2017-07-26T05:35:05.673Z] [INFO] Cluster Client Created
[2017-07-26T05:35:05.703Z] [INFO] Kubernetes version: v1.6.1+5115d708d7
[2017-07-26T05:35:05.703Z] [INFO] == REGISTRY CX == 
[2017-07-26T05:35:05.703Z] [INFO] Name: rhcc
[2017-07-26T05:35:05.703Z] [INFO] Url: registry.access.stage.redhat.com
[2017-07-26T05:35:05.704Z] [INFO] Listening for provision messages
[2017-07-26T05:35:06.021Z] [INFO] Initiating Recovery Process
[2017-07-26T05:35:06.024Z] [INFO] Recovery complete
[2017-07-26T05:35:06.024Z] [NOTICE] recover called
[2017-07-26T05:35:06.024Z] [INFO] Broker configured to bootstrap on startup
[2017-07-26T05:35:06.024Z] [INFO] Attempting bootstrap...
[2017-07-26T05:35:06.024Z] [INFO] AnsibleBroker::Bootstrap
[2017-07-26T05:35:07.671Z] [INFO] Didn't find encoded Spec label. Assuming image is not APB and skipping.
[2017-07-26T05:35:07.982Z] [NOTICE] Broker successfully bootstrapped on startup
[2017-07-26T05:35:07.982Z] [NOTICE] Ansible Service Broker Started
[2017-07-26T05:35:07.982Z] [NOTICE] Listening on http://0.0.0.0:1338
[2017-07-26T05:36:42.109Z] [INFO] ASYNC provisioning in progress
[2017-07-26T05:36:42.109Z] [NOTICE] ============================================================
[2017-07-26T05:36:42.109Z] [NOTICE]                        PROVISIONING                         
[2017-07-26T05:36:42.109Z] [NOTICE] ============================================================
[2017-07-26T05:36:42.109Z] [NOTICE] Spec.Id: 4fbcc051-03af-40c7-86e2-dc12977e6b5d
[2017-07-26T05:36:42.109Z] [NOTICE] Spec.Name: mediawiki-apb
[2017-07-26T05:36:42.11Z] [NOTICE] Spec.Image: openshift3/mediawiki-apb
[2017-07-26T05:36:42.11Z] [NOTICE] Spec.Description: Mediawiki123 apb implementation
[2017-07-26T05:36:42.11Z] [NOTICE] ============================================================
[2017-07-26T05:36:42.11Z] [INFO] Checking if project dma exists...
10.128.0.1 - - [26/Jul/2017:05:36:42 +0000] "PUT /v2/service_instances/2cd33d57-491a-42d0-9bf9-979b5e136f66?accepts_incomplete=true HTTP/1.1" 202 58
[2017-07-26T05:36:42.409Z] [ERROR] Project dma does NOT exist! Cannot provision requested mediawiki-apb
[2017-07-26T05:36:42.409Z] [ERROR] broker::Provision error occurred.
[2017-07-26T05:36:42.409Z] [ERROR] Project dma does not exist
[2017-07-26T05:36:42.409Z] [ERROR] Attempting to destroy APB sandbox if it has been created
[2017-07-26T05:36:42.409Z] [INFO] Requested destruction of APB sandbox with empty handle, skipping.
[2017-07-26T05:37:22.982Z] [INFO] ASYNC provisioning in progress
[2017-07-26T05:37:22.982Z] [NOTICE] ============================================================
[2017-07-26T05:37:22.982Z] [NOTICE]                        PROVISIONING                         
[2017-07-26T05:37:22.982Z] [NOTICE] ============================================================
[2017-07-26T05:37:22.982Z] [NOTICE] Spec.Id: 4fbcc051-03af-40c7-86e2-dc12977e6b5d
[2017-07-26T05:37:22.982Z] [NOTICE] Spec.Name: mediawiki-apb
[2017-07-26T05:37:22.982Z] [NOTICE] Spec.Image: openshift3/mediawiki-apb
[2017-07-26T05:37:22.982Z] [NOTICE] Spec.Description: Mediawiki123 apb implementation
[2017-07-26T05:37:22.982Z] [NOTICE] ============================================================
[2017-07-26T05:37:22.982Z] [INFO] Checking if project dma exists...
10.128.0.1 - - [26/Jul/2017:05:37:22 +0000] "PUT /v2/service_instances/1ceebfe5-ee94-4d14-8b38-c214dd3e12fc?accepts_incomplete=true HTTP/1.1" 202 58
[2017-07-26T05:37:23.194Z] [INFO] Successfully wrote resources to /tmp/asb-resource-files/apb-aacf11b1-da99-4d5f-94ea-5d114354822e.yaml
[2017-07-26T05:37:23.514Z] [INFO] Successfully created apb sandbox: [ apb-aacf11b1-da99-4d5f-94ea-5d114354822e ]
[2017-07-26T05:37:23.514Z] [NOTICE] Creating pod "apb-aacf11b1-da99-4d5f-94ea-5d114354822e" in the dma namespace
[2017-07-26T05:37:29.744Z] [INFO] Container not up yet, retrying 1 of 150 on pod apb-aacf11b1-da99-4d5f-94ea-5d114354822e
[2017-07-26T05:38:03.221Z] [INFO] Destroying APB sandbox...


Note You need to log in before you can comment on or make changes to this bug.