Bug 1786037

Summary: Opsrc is not available in OSP 13 with kuryr
Product: OpenShift Container Platform Reporter: Fan Jia <jfan>
Component: OLMAssignee: Kevin Rizza <krizza>
OLM sub component: OperatorHub QA Contact: Fan Jia <jfan>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: medium CC: anli, bandrade, chezhang, chuo, dageoffr, dyan, jiazha, krizza, scolange, xjiang, xtian
Version: 4.3.0   
Target Milestone: ---   
Target Release: 4.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-05-04 11:20:55 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1811131    
Bug Blocks:    

Description Fan Jia 2019-12-23 07:45:48 UTC
Description of problem:
The env "OSP 13 with kuryr" is "OSP octavia load balancer service does not support UDP". And the marketplace keep "Downloading metadata from Namespace [redhat-operators] of [https://quay.io/cnr]".

Version-Release number of selected component (if applicable):
4.3.0-0.nightly-2019-12-22-054650

How reproducible:
always

Steps to Reproduce:
The default opsrc

Actual results:
1. # oc get opsrc
NAME                  TYPE          ENDPOINT              REGISTRY              DISPLAYNAME           PUBLISHER   STATUS        MESSAGE                                                                                                                                                                       AGE
certified-operators   appregistry   https://quay.io/cnr   certified-operators   Certified Operators   Red Hat     Configuring   Get https://quay.io/cnr/api/v1/packages?namespace=certified-operators: dial tcp: lookup quay.io on 172.30.0.10:53: read udp 10.128.74.11:55025->172.30.0.10:53: i/o timeout   10m
community-operators   appregistry   https://quay.io/cnr   community-operators   Community Operators   Red Hat     Configuring   Get https://quay.io/cnr/api/v1/packages?namespace=community-operators: context deadline exceeded                                                                              10m
redhat-operators      appregistry   https://quay.io/cnr   redhat-operators      Red Hat Operators     Red Hat     Configuring   Get https://quay.io/cnr/api/v1/packages?namespace=redhat-operators: context deadline exceeded                                                                                 10m

2. oc logs marketplace-operator-xxxx
`
time="2019-12-23T07:35:48Z" level=info msg="Reconciling OperatorSource openshift-marketplace/community-operators\n"
time="2019-12-23T07:35:48Z" level=info msg="Downloading metadata from Namespace [community-operators] of [https://quay.io/cnr]" name=community-operators namespace=openshift-marketplace type=OperatorSource
time="2019-12-23T07:36:19Z" level=info msg="Reconciling OperatorSource openshift-marketplace/certified-operators\n"
time="2019-12-23T07:36:19Z" level=info msg="Downloading metadata from Namespace [certified-operators] of [https://quay.io/cnr]" name=certified-operators namespace=openshift-marketplace type=OperatorSource
time="2019-12-23T07:36:50Z" level=info msg="Reconciling OperatorSource openshift-marketplace/redhat-operators\n"
time="2019-12-23T07:36:50Z" level=info msg="Downloading metadata from Namespace [redhat-operators] of [https://quay.io/cnr]" name=redhat-operators namespace=openshift-marketplace type=OperatorSource
time="2019-12-23T07:37:21Z" level=info msg="Reconciling OperatorSource openshift-marketplace/community-operators\n"
time="2019-12-23T07:37:21Z" level=info msg="Downloading metadata from Namespace [community-operators] of [https://quay.io/cnr]" name=community-operators namespace=openshift-marketplace type=OperatorSource
time="2019-12-23T07:37:52Z" level=info msg="Reconciling OperatorSource openshift-marketplace/certified-operators\n"
time="2019-12-23T07:37:52Z" level=info msg="Downloading metadata from Namespace [certified-operators] of [https://quay.io/cnr]" name=certified-operators namespace=openshift-marketplace type=OperatorSource
time="2019-12-23T07:38:23Z" level=info msg="Reconciling OperatorSource openshift-marketplace/redhat-operators\n"
time="2019-12-23T07:38:23Z" level=info msg="Downloading metadata from Namespace [redhat-operators] of [https://quay.io/cnr]" name=redhat-operators namespace=openshift-marketplace type=OperatorSource

`

Expected results:
1. Opsrc is Success and load the right packages.

Additional info:

Comment 2 Dan Geoffroy 2020-01-03 14:47:38 UTC
Moving this to 4.4 where we will continue to investigate it.  If verified we will consider for 4.3.z backport.

Comment 3 Kevin Rizza 2020-01-07 13:38:47 UTC
Hi Jia,

Based on the logs, this cluster appears to have some funky stuff going in with networking. That error at a high level means that the marketplace operator is not able to query the external quay.io app-registry endpoint. Based on the cluster description, it sounds like there is some unique configuration. Could you tell us what is unique about this config? In order for the marketplace to work the marketplace operator needs to have network access to make HTTP GET requests to https://quay.io/cnr, and if it does not then this is expected.

Comment 5 Fan Jia 2020-01-08 01:50:36 UTC
(In reply to Kevin Rizza from comment #3)
> Hi Jia,
> 
> Based on the logs, this cluster appears to have some funky stuff going in
> with networking. That error at a high level means that the marketplace
> operator is not able to query the external quay.io app-registry endpoint.
> Based on the cluster description, it sounds like there is some unique
> configuration. Could you tell us what is unique about this config? In order
> for the marketplace to work the marketplace operator needs to have network
> access to make HTTP GET requests to https://quay.io/cnr, and if it does not
> then this is expected.

Hi kevin, this cluster doesn't support UDP link: "OSP octavia load balancer service does not support UDP"

Comment 6 Fan Jia 2020-01-15 06:11:35 UTC
Can't be tested since no 4.4 kury env for now.

Comment 7 Fan Jia 2020-03-19 03:17:43 UTC
env:
cv:4.4.0-0.nightly-2020-03-18-011500
OSP 16 with kury

test result:
#oc get opsrc -n openshift-marketplace
NAME                  TYPE          ENDPOINT              REGISTRY              DISPLAYNAME           PUBLISHER   STATUS      MESSAGE                                       AGE
certified-operators   appregistry   https://quay.io/cnr   certified-operators   Certified Operators   Red Hat     Succeeded   The object has been successfully reconciled   18h
community-operators   appregistry   https://quay.io/cnr   community-operators   Community Operators   Red Hat     Succeeded   The object has been successfully reconciled   18h
qe-app-registry       appregistry   https://quay.io/cnr   aosqe4                                      Red Hat     Succeeded   The object has been successfully reconciled   16m
redhat-marketplace    appregistry   https://quay.io/cnr   redhat-marketplace    Red Hat Marketplace   Red Hat     Succeeded   The object has been successfully reconciled   18h
redhat-operators      appregistry   https://quay.io/cnr   redhat-operators      Red Hat Operators     Red Hat     Succeeded   The object has been successfully reconciled   18h

#oc get pods -n openshift-marketplace
NAME                                    READY   STATUS    RESTARTS   AGE
certified-operators-66bc7c8c65-f68q2    1/1     Running   0          10h
community-operators-6f5dc697dd-z4r9k    1/1     Running   0          12h
marketplace-operator-75fdff9554-6f2pv   1/1     Running   0          18h
qe-app-registry-7b554cbdc9-5tvvx        1/1     Running   0          16m
redhat-marketplace-847bff48dc-mh5dz     1/1     Running   0          18h
redhat-operators-7757fd88db-slwfh       1/1     Running   0          18h

Comment 9 errata-xmlrpc 2020-05-04 11:20:55 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581