Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1779101

Summary: asb/tsb can not be installed on baremental disconnected cluster with FIPS on
Product: OpenShift Container Platform Reporter: Cuiping HUO <chuo>
Component: Service BrokerAssignee: amacdona <austin>
Status: CLOSED ERRATA QA Contact: Cuiping HUO <chuo>
Severity: high Docs Contact:
Priority: high    
Version: 4.3.0CC: aos-bugs, austin, chezhang, jesusr, jfan, jiazha
Target Milestone: ---   
Target Release: 4.3.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-01-23 11:14:59 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
a detailed log of asb/tsb operator none

Description Cuiping HUO 2019-12-03 10:14:09 UTC 
Created attachment 1641629 [details]
a detailed log of asb/tsb operator

Description of problem:
asb/tsb can not be installed on baremental disconnected cluster with FIPS on

Version-Release number of selected component (if applicable):
4.3.0-0.nightly-2019-11-29-051144
asb csv:openshiftansibleservicebroker.4.3.0-201911220712
tsb csv:openshifttemplateservicebrokeroperator.4.3.0-201911220712
How reproducible:
Always

Steps to Reproduce:
1.spin up a baremental disconnected cluster with FIPs on 
2.install asb operator 
3.install asb
4.install tsb operator
5.install tsb

Actual results:
asb/tsb operator install sucessfully, but asb and tsb failed with error 
message: 'An unhandled exception occurred while running the lookup plugin ''k8s''.
      Error was a <type ''exceptions.ValueError''>, original message: error:060800A3:digital
      envelope routines:EVP_DigestInit_ex:disabled for fips'


Expected results:
asb/tsb operator and asb/tsb should all install sucessfully

Additional info:
$ oc get automationbroker ansible-service-broker -o yaml
apiVersion: osb.openshift.io/v1
kind: AutomationBroker
metadata:
  creationTimestamp: "2019-12-03T10:00:53Z"
  finalizers:
  - finalizer.osb.openshift.io
  generation: 1
  name: ansible-service-broker
  namespace: openshift-ansible-service-broker
  resourceVersion: "52556"
  selfLink: /apis/osb.openshift.io/v1/namespaces/openshift-ansible-service-broker/automationbrokers/ansible-service-broker
  uid: ae0b74de-b5ca-47d6-b2c4-0e9d7c21abe8
spec:
  createBrokerNamespace: "false"
  registries:
  - auth_name: asb-registry-auth
    auth_type: secret
    name: rhcc
    type: rhcc
    url: https://registry.redhat.io
    white_list:
    - .*-apb$
  waitForBroker: "false"
status:
  conditions:
  - lastTransitionTime: "2019-12-03T10:00:53Z"
    message: Running reconciliation
    reason: Running
    status: "False"
    type: Running
  - ansibleResult:
      changed: 0
      completion: 2019-12-03T10:00:55.586046
      failures: 1
      ok: 0
      skipped: 0
    lastTransitionTime: "2019-12-03T10:00:56Z"
    message: 'An unhandled exception occurred while running the lookup plugin ''k8s''.
      Error was a <type ''exceptions.ValueError''>, original message: error:060800A3:digital
      envelope routines:EVP_DigestInit_ex:disabled for fips'
    reason: Failed
    status: "True"
    type: Failure

$ oc  get templateservicebroker template-service-broker -o yaml
apiVersion: osb.openshift.io/v1
kind: TemplateServiceBroker
metadata:
  creationTimestamp: "2019-12-03T10:10:25Z"
  finalizers:
  - finalizer.osb.openshift.io
  generation: 1
  name: template-service-broker
  namespace: openshift-template-service-broker
  resourceVersion: "55932"
  selfLink: /apis/osb.openshift.io/v1/namespaces/openshift-template-service-broker/templateservicebrokers/template-service-broker
  uid: 1e5da75e-0c81-4672-b766-7c90f1e0a06e
spec: {}
status:
  conditions:
  - lastTransitionTime: "2019-12-03T10:10:24Z"
    message: Running reconciliation
    reason: Running
    status: "False"
    type: Running
  - ansibleResult:
      changed: 0
      completion: 2019-12-03T10:10:26.807339
      failures: 1
      ok: 2
      skipped: 0
    lastTransitionTime: "2019-12-03T10:10:27Z"
    message: 'An unhandled exception occurred while running the lookup plugin ''k8s''.
      Error was a <type ''exceptions.ValueError''>, original message: error:060800A3:digital
      envelope routines:EVP_DigestInit_ex:disabled for fips'
    reason: Failed
    status: "True"
    type: Failure
Comment 2 Jesus M. Rodriguez 2019-12-09 20:58:35 UTC 
This has been fixed in the following PRs for the different releases found for python-restclient-openshift:

master: https://github.com/openshift/openshift-restclient-python/pull/342
release-0.10: https://github.com/openshift/openshift-restclient-python/pull/345
release-0.9: https://github.com/openshift/openshift-restclient-python/pull/343
release-0.8: https://github.com/openshift/openshift-restclient-python/pull/344

The downstream ansible-operator uses a 0.8 based release:
http://pkgs.devel.redhat.com/cgit/rpms/python-openshift/tree/python-openshift.spec?h=rhaos-4.2-asb-rhel-7#n19

Working on getting 0.8.11 built in brew from the latest upstream source.
Comment 3 Jesus M. Rodriguez 2019-12-10 19:33:43 UTC 
python-openshift built in brew https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1032900
Comment 4 Jesus M. Rodriguez 2019-12-11 02:31:51 UTC 
4.3 images now have 0.8.11 http://download-node-02.eng.bos.redhat.com/rcm-guest/puddles/RHAOS/AtomicOpenShift/4.3/2019-12-10.7/x86_64/os/Packages/
Comment 6 Cuiping HUO 2019-12-12 07:12:34 UTC 
Verification failed.
cluster version: 4.3.0-0.nightly-2019-12-12-004325(with FIPs on)


$ oc get automationbroker ansible-service-broker -o yaml
apiVersion: osb.openshift.io/v1
kind: AutomationBroker
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"osb.openshift.io/v1","kind":"AutomationBroker","metadata":{"annotations":{},"name":"ansible-service-broker","namespace":"openshift-ansible-service-broker"},"spec":{"createBrokerNamespace":"false","registries":[{"images":["openshift/mediawiki-apb","openshift/postgresql-apb","openshift/mariadb-apb","openshift/mysql-apb"],"name":"test","skip_verify_tls":true,"tag":"latest","type":"local_openshift","url":"image-registry.openshift-image-registry.svc:5000","white_list":[".*-apb$"]}],"waitForBroker":"false"}}
  creationTimestamp: "2019-12-12T06:17:37Z"
  finalizers:
  - finalizer.osb.openshift.io
  generation: 1
  name: ansible-service-broker
  namespace: openshift-ansible-service-broker
  resourceVersion: "50513"
  selfLink: /apis/osb.openshift.io/v1/namespaces/openshift-ansible-service-broker/automationbrokers/ansible-service-broker
  uid: adb012b5-c389-4022-85f4-15d26088f38e
spec:
  createBrokerNamespace: "false"
  registries:
  - images:
    - openshift/mediawiki-apb
    - openshift/postgresql-apb
    - openshift/mariadb-apb
    - openshift/mysql-apb
    name: test
    skip_verify_tls: true
    tag: latest
    type: local_openshift
    url: image-registry.openshift-image-registry.svc:5000
    white_list:
    - .*-apb$
  waitForBroker: "false"
status:
  conditions:
  - lastTransitionTime: "2019-12-12T06:17:37Z"
    message: Running reconciliation
    reason: Running
    status: "False"
    type: Running
  - ansibleResult:
      changed: 0
      completion: 2019-12-12T06:17:39.350372
      failures: 1
      ok: 0
      skipped: 0
    lastTransitionTime: "2019-12-12T06:17:39Z"
    message: 'An unhandled exception occurred while running the lookup plugin ''k8s''.
      Error was a <type ''exceptions.ValueError''>, original message: error:060800A3:digital
      envelope routines:EVP_DigestInit_ex:disabled for fips'
    reason: Failed
    status: "True"
    type: Failure
$ oc get templateservicebroker template-service-broker -o yaml
apiVersion: osb.openshift.io/v1
kind: TemplateServiceBroker
metadata:
  creationTimestamp: "2019-12-12T06:51:56Z"
  finalizers:
  - finalizer.osb.openshift.io
  generation: 1
  name: template-service-broker
  namespace: openshift-template-service-broker
  resourceVersion: "61308"
  selfLink: /apis/osb.openshift.io/v1/namespaces/openshift-template-service-broker/templateservicebrokers/template-service-broker
  uid: e80191df-528b-400a-bf1e-cd6146ba5489
spec: {}
status:
  conditions:
  - lastTransitionTime: "2019-12-12T06:51:56Z"
    message: Running reconciliation
    reason: Running
    status: "False"
    type: Running
  - ansibleResult:
      changed: 0
      completion: 2019-12-12T06:51:58.087002
      failures: 1
      ok: 2
      skipped: 0
    lastTransitionTime: "2019-12-12T06:51:58Z"
    message: 'An unhandled exception occurred while running the lookup plugin ''k8s''.
      Error was a <type ''exceptions.ValueError''>, original message: error:060800A3:digital
      envelope routines:EVP_DigestInit_ex:disabled for fips'
    reason: Failed
    status: "True"
    type: Failure
$ oc get csv -n openshift-ansible-service-broker
NAME                                               DISPLAY                                     VERSION              REPLACES   PHASE
openshiftansibleservicebroker.4.3.0-201912111446   OpenShift Ansible Service Broker Operator   4.3.0-201912111446              Succeeded

$ oc get csv -n openshift-template-service-broker
NAME                                                        DISPLAY                                      VERSION              REPLACES   PHASE
openshifttemplateservicebrokeroperator.4.3.0-201912111317   OpenShift Template Service Broker Operator   4.3.0-201912111317              Succeeded

$  oc image info registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-ansible-service-broker-operator@sha256:2cc5ea1bf3cb7fb6120c5e539f4301316300abad77a338ca68914ae200a39aa8| grep commit
             io.openshift.build.commit.id=346a81a77323baeb9f8bcb13437f7e7e32a0824f
             io.openshift.build.commit.url=https://github.com/openshift/ansible-service-broker/commit/346a81a77323baeb9f8bcb13437f7e7e32a0824f

$ oc image info registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-template-service-broker-operator@sha256:c713abb87d5b022b3251a0df35403adef7e4af8e0a8b3a2258bf810d6c087e8a | grep commit
             io.openshift.build.commit.id=9051edc49e08a20a63c8a3282df4758c25d0e27a
             io.openshift.build.commit.url=https://github.com/openshift/template-service-broker-operator/commit/9051edc49e08a20a63c8a3282df4758c25d0e27a
Comment 7 Jesus M. Rodriguez 2019-12-12 22:00:53 UTC 
The problem was there was a build problem and the base image wasn't rebuilt.  You will need at least v4.3.0-201912121330 of the ansible operator. 

[jesusr@transam 0718]$ docker run -it  --network host --entrypoint=/bin/bash registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-ansible-service-broker-operator:v4.3.0-201912121330
Unable to find image 'registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-ansible-service-broker-operator:v4.3.0-201912121330' locally
v4.3.0-201912121330: Pulling from rh-osbs/openshift-ose-ansible-service-broker-operator
d327c1598329: Already exists 
48ed3bfd8226: Already exists 
f912f2abfc7d: Already exists 
0c844c72e567: Already exists 
95928b9306a8: Already exists 
9f3d80e1255f: Pull complete 
Digest: sha256:e233a7075247105956fd9d7e2fdba072663bfad6cb7cb55fee662913569fbb04
Status: Downloaded newer image for registry-proxy.engineering.redhat.com/rh-osbs/openshift-ose-ansible-service-broker-operator:v4.3.0-201912121330
bash-4.2$ rpm -q python2-openshift
python2-openshift-0.8.11-1.el7.noarch
bash-4.2$
Comment 9 Cuiping HUO 2019-12-13 06:45:39 UTC 
Verified.
cluster version: 4.3.0-0.nightly-2019-12-12-155629(with FIPs on)

$ oc get automationbroker ansible-service-broker -n openshift-ansible-service-broker -o yaml
apiVersion: v1
items:
- apiVersion: osb.openshift.io/v1
  kind: AutomationBroker
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"osb.openshift.io/v1","kind":"AutomationBroker","metadata":{"annotations":{},"name":"ansible-service-broker","namespace":"openshift-ansible-service-broker"},"spec":{"createBrokerNamespace":"false","registries":[{"images":["openshift/mediawiki-apb","openshift/postgresql-apb","openshift/mariadb-apb","openshift/mysql-apb"],"name":"test","skip_verify_tls":true,"tag":"latest","type":"local_openshift","url":"image-registry.openshift-image-registry.svc:5000","white_list":[".*-apb$"]}],"waitForBroker":"false"}}
    creationTimestamp: "2019-12-13T05:48:21Z"
    finalizers:
    - finalizer.osb.openshift.io
    generation: 1
    name: ansible-service-broker
    namespace: openshift-ansible-service-broker
    resourceVersion: "76501"
    selfLink: /apis/osb.openshift.io/v1/namespaces/openshift-ansible-service-broker/automationbrokers/ansible-service-broker
    uid: 268f1c4d-ce81-414c-a4c7-ebaea7e6d5c1
  spec:
    createBrokerNamespace: "false"
    registries:
    - images:
      - openshift/mediawiki-apb
      - openshift/postgresql-apb
      - openshift/mariadb-apb
      - openshift/mysql-apb
      name: test
      skip_verify_tls: true
      tag: latest
      type: local_openshift
      url: image-registry.openshift-image-registry.svc:5000
      white_list:
      - .*-apb$
    waitForBroker: "false"
  status:
    conditions:
    - ansibleResult:
        changed: 0
        completion: 2019-12-13T06:43:54.477074
        failures: 0
        ok: 15
        skipped: 11
      lastTransitionTime: "2019-12-13T05:53:59Z"
      message: Awaiting next reconciliation
      reason: Successful
      status: "True"
      type: Running
    reconciledGeneration: "1"
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

$ oc get templateservicebroker template-service-broker -n openshift-template-service-broker -o yaml
apiVersion: v1
items:
- apiVersion: osb.openshift.io/v1
  kind: TemplateServiceBroker
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"osb.openshift.io/v1","kind":"TemplateServiceBroker","metadata":{"annotations":{},"name":"template-service-broker","namespace":"openshift-template-service-broker"},"spec":{}}
    creationTimestamp: "2019-12-13T05:53:16Z"
    finalizers:
    - finalizer.osb.openshift.io
    generation: 1
    name: template-service-broker
    namespace: openshift-template-service-broker
    resourceVersion: "71073"
    selfLink: /apis/osb.openshift.io/v1/namespaces/openshift-template-service-broker/templateservicebrokers/template-service-broker
    uid: 1e1222e5-eb3a-49c4-8e1b-65159d201042
  spec: {}
  status:
    conditions:
    - ansibleResult:
        changed: 0
        completion: 2019-12-13T06:27:10.313865
        failures: 0
        ok: 6
        skipped: 0
      lastTransitionTime: "2019-12-13T06:10:39Z"
      message: Awaiting next reconciliation
      reason: Successful
      status: "True"
      type: Running
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""
$ oc get csv -n openshift-ansible-service-broker
NAME                                               DISPLAY                                     VERSION              REPLACES   PHASE
openshiftansibleservicebroker.4.3.0-201912121917   OpenShift Ansible Service Broker Operator   4.3.0-201912121917              Succeeded
$ oc get csv -n openshift-template-service-broker
NAME                                                        DISPLAY                                      VERSION              REPLACES   PHASE
openshifttemplateservicebrokeroperator.4.3.0-201912122317   OpenShift Template Service Broker Operator   4.3.0-201912122317              Succeeded
Comment 11 errata-xmlrpc 2020-01-23 11:14:59 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0062