Bug 1705779

Summary: AutomationBroker created successfully but asb-* pods do not start - permission errors in automation operator logs
Product: OpenShift Container Platform Reporter: Mike Fiedler <mifiedle>
Component: Service BrokerAssignee: Shawn Hurley <shurley>
Status: CLOSED DUPLICATE QA Contact: Zhang Cheng <chezhang>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 4.1.0CC: aos-bugs
Target Milestone: ---Keywords: TestBlocker
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-05-03 12:50:06 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1685458    
Attachments:
Description Flags
automation broker operator pod logs after failed install none

Description Mike Fiedler 2019-05-03 00:10:30 UTC
Created attachment 1562040 [details]
automation broker operator pod logs after failed install

Description of problem:

Tried to install ASB using the directions in the 4.1 documentation:   https://docs.openshift.com/container-platform/4.1/applications/service_brokers/installing-ansible-service-broker.html

My steps were:

0. Change service catalog operators to Managed
1. Install the Automation Broker Operator to the openshift-ansible-service-broker namespace (Catalog->OperatorHub->Automation Broker Operator->Install

# oc get pods
NAME                                          READY   STATUS    RESTARTS   AGE
automation-broker-operator-5d769bcbb5-ghz9f   1/1     Running   0          40s

2. Create the clusterrolebinding:

  - Navigate to Administration → Role Bindings and click Create Binding.
  - For the Binding Type, select Cluster-wide Role Binding (ClusterRoleBinding).
  - For the Role Binding, enter ansible-service-broker in the Name field.
  - For the Role, select admin.
  - For the Subject, choose the Service Account option, select the openshift-ansible-service-broker namespace, and enter openshift-ansible-service-broker-operator in the Subject Name field.

3. Create the AutomationBroker CR from the console
  - Installed Operators -> Automation Broker -> Create New.   
  - ensure it is installed to the openshift-ansible-service-broker namespace

At this point the automationbroker CR is created and shows Running:

apiVersion: v1
items:
- apiVersion: osb.openshift.io/v1alpha1
  kind: AutomationBroker
  metadata:
    creationTimestamp: "2019-05-03T00:04:09Z"
    finalizers:
    - finalizer.osb.openshift.io
    generation: 1
    name: ansible-service-broker
    namespace: openshift-ansible-service-broker
    resourceVersion: "158870"
    selfLink: /apis/osb.openshift.io/v1alpha1/namespaces/openshift-ansible-service-broker/automationbrokers/ansible-service-broker
    uid: f96b03d0-6d36-11e9-999b-02141e0ccc98
  spec:
    createBrokerNamespace: "false"
    waitForBroker: "false"
  status:
    conditions:
    - ansibleResult:
        changed: 0
        completion: 2019-05-03T00:04:11.554527
        failures: 0
        ok: 5
        skipped: 4
      lastTransitionTime: "2019-05-03T00:04:09Z"
      message: Awaiting next reconciliation
      reason: Successful
      status: "True"
      type: Running



But the asb-pods are never created.


The automation-broker-pod log shows a fatal error related to permissions:

fatal: [localhost]: UNREACHABLE! => {\"changed\": false, \"msg\": \"Authentication or permission failure. In some cases, you may have been able to authenticate and did not have permissions on the target directory. Consider changing the remote tmp path in ansible.cfg to a path rooted in \\\"/tmp\\\". Failed command was: ( umask 77 && mkdir -p \\\"` echo /.ansible/tmp/ansible-tmp-1556841851.54-95948707781686 `\\\" && echo ansible-tmp-1556841851.54-95948707781686=\\\"` echo /.ansible/tmp/ansible-tmp-1556841851.54-95948707781686 `\\\" ), exited with result 1\", \"unreachable\": true}\u001b[0m\n\r\n


Version-Release number of selected component (if applicable):  https://openshift-release.svc.ci.openshift.org/releasestream/4.1.0-0.nightly/release/4.1.0-0.nightly-2019-05-02-190618


How reproducible: Always

Comment 2 Shawn Hurley 2019-05-03 12:50:06 UTC
The manifestation is slightly different for the two bugs but the underlying caused is fixed by the same PR.

*** This bug has been marked as a duplicate of bug 1705319 ***

Comment 3 Mike Fiedler 2019-05-03 14:17:36 UTC
Verified this works OK on 4.1.0-0.nightly-2019-05-03-093152 which has the fix for bug 1705319.   Thanks.