Bug 1686668

Summary: [marketplace] Operator Hub Removing installed operators from a custom AppRegistry results in invalid install CSC
Product: OpenShift Container Platform Reporter: Sami Shahin <sashahin>
Component: OLMAssignee: Kevin Rizza <krizza>
Status: CLOSED ERRATA QA Contact: Fan Jia <jfan>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.1.0CC: aravindh, jfan
Target Milestone: ---   
Target Release: 4.1.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-06-04 10:45:20 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Sami Shahin 2019-03-08 01:04:16 UTC
Description of problem:
After removing and adding operators from a custom app registry and forcing a resync, installing a new operator results in the installation CSC getting stuck in the "Configuring" phase with the message "Still resolving package(s) - . Please make sure these are valid packages."

Version-Release number of selected component (if applicable):
ClusterVersion: quay.io/openshift-release-dev/ocp-release@sha256:641c0e4f550af59ec20349187a31751ae5108270f13332d1771935520ebf34c1
Installer version: 0.14.0

How reproducible:


Steps to Reproduce:
1. Create a namespace on quay, and add at least one Application to it with an operator bundle. This can be done with the `operator-courier push` command as outlined here: https://github.com/operator-framework/operator-courier

Be sure to note the supported installMode for this operator. I used an AllNamespace operator.

2. Add a custom OperatorSource CR to the openshift-marketplace namespace pointing to your quay namespace:

apiVersion: marketplace.redhat.com/v1alpha1
kind: OperatorSource
metadata:
  name: custom-operators
  namespace: openshift-marketplace
spec:
  displayName: Custom Operators
  endpoint: https://quay.io/cnr
  publisher: Red Hat
  registryNamespace: <YOUR_QUAY_NAMESPACE>
  type: appregistry

3. Install the operator via the OpenShift console UI.

4. After the operator is fully installed (subscription should say "Up to date" under "Operator Management" in the "Operator Subscriptions" tab) delete the Application from quay (click on the application and go to "Settings", at the bottom of the page there should be a red "Delete Application" button).

5. Upload a different operator to the quay namespace. The name should be different from the original. Also, the new operator must support the same installModes as the previous one. In my case it was AllNamespaces.

6. Delete and reapply the OperatorSource to force a resync.

7. From the OperatorHub UI, install the new operator that was pushed to quay.

8. Under "Operator Management" in the "Operator Subscriptions" tab, remove the subscription for the OLD operator. Make sure the checkbox is checked.

9. From the CLI, run:

$ oc get csc -n openshift-marketplace installed-custom-openshift-operators -o yaml

The "openshift-operators" part of the CSC name comes from the installed namespace. If using an AllNamespace install operator it should match the command provided.

Actual results:

$ oc get csc -n openshift-marketplace installed-custom-openshift-operators -o yaml
apiVersion: marketplace.redhat.com/v1alpha1
kind: CatalogSourceConfig
metadata:
  creationTimestamp: 2019-03-07T20:05:35Z
  finalizers:
  - finalizer.catalogsourceconfigs.marketplace.redhat.com
  generation: 1
  name: installed-custom-openshift-operators
  namespace: openshift-marketplace
  resourceVersion: "1036062"
  selfLink: /apis/marketplace.redhat.com/v1alpha1/namespaces/openshift-marketplace/catalogsourceconfigs/installed-custom-openshift-operators
  uid: 5e6e3cb5-4114-11e9-b9f9-0a13676e199a
spec:
  csDisplayName: Custom Operators
  csPublisher: Custom
  packages: ',strimzi-kafka-operator-test'
  targetNamespace: openshift-operators
status:
  currentPhase:
    lastTransitionTime: 2019-03-08T00:21:18Z
    lastUpdateTime: 2019-03-08T00:21:18Z
    phase:
      message: Still resolving package(s) - . Please make sure these are valid packages.
      name: Configuring

The CSC is stuck in the Configuring phase. Notice the `spec.packages` string has a leading comma before the operator's name (which is the second operator installed).

Expected results:
$ oc get csc -n openshift-marketplace installed-custom-openshift-operators -o yaml
apiVersion: marketplace.redhat.com/v1alpha1
kind: CatalogSourceConfig
metadata:
  creationTimestamp: 2019-03-08T00:26:30Z
  finalizers:
  - finalizer.catalogsourceconfigs.marketplace.redhat.com
  generation: 1
  name: installed-custom-openshift-operators
  namespace: openshift-marketplace
  resourceVersion: "1039455"
  selfLink: /apis/marketplace.redhat.com/v1alpha1/namespaces/openshift-marketplace/catalogsourceconfigs/installed-custom-openshift-operators
  uid: d1ef5829-4138-11e9-89b8-0297441e13f2
spec:
  csDisplayName: Custom Operators
  csPublisher: Custom
  packages: strimzi-kafka-operator-test
  targetNamespace: openshift-operators
status:
  currentPhase:
    lastTransitionTime: 2019-03-08T00:26:30Z
    lastUpdateTime: 2019-03-08T00:26:30Z
    phase:
      message: The object has been successfully reconciled
      name: Succeeded

The CSC succeeds.

Additional info:

Comment 1 Fan Jia 2019-03-08 05:09:19 UTC
Sami, the remove of "subscription" won't delete the package in the corresponding CSC. But of course the delete of package in the remote registory should cause the CSC's Reconciling (delete the package). And I report it as https://bugzilla.redhat.com/show_bug.cgi?id=1686707,  Thank you.

Comment 2 Fan Jia 2019-03-08 05:13:39 UTC
*** Bug 1686707 has been marked as a duplicate of this bug. ***

Comment 3 Sami Shahin 2019-03-08 05:28:50 UTC
Hi Jia,

I realize deleting the subscription doesn't remove a package from the CSC, just listing all my steps. :)

After looking at the bug linked, I don't think this is a duplicate. The issue here isn't that the CSC isn't synced up, since I'm forcing a resync by deleting and reapplying the OperatorSource. The issue is with how the install CSC is resynced when operators are removed from the custom app registry. The bug linked notes an issue with syncing the install CSC with the app registry at all, based on my interpretation. They may be related, but the symptoms are distinct.

Comment 4 Kevin Rizza 2019-03-21 12:36:58 UTC
This pr addresses the described issue: https://github.com/operator-framework/operator-marketplace/pull/138

Comment 5 Fan Jia 2019-03-22 03:10:37 UTC
The latest nightly build doesn't include this pr , will confirm after the pr is included by nightly build
cv: 4.0.0-0.nightly-2019-03-22-002648
marketplace image:quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c802c1681883549a6a6b123719866b382218294259d55d5464415cf9464b189f
marketplace commit:68b01f6507267f6deefa952a1034b74a5e8480bf

Comment 6 Fan Jia 2019-03-22 11:56:31 UTC
(In reply to Sami Shahin from comment #3)

Yes , Sami, and the bug I mentioned is fixed now.

Comment 7 Fan Jia 2019-03-22 12:24:38 UTC
test env:
cv: 4.0.0-0.nightly-2019-03-22-062943
marketplace image:quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5782d66f4f51e925a20832085e7e1457e35400543449e83c56b4213bf69725d4
marketplace commit:213899cee5dcf37f78522f56196a860be10b1e94

The csc still has the comma before the operator's name (which is the second operator installed, delete the first operator in the quay). But the csc ignore the invalid values , so the csc can be created success and the pod registory is created success.

Comment 9 errata-xmlrpc 2019-06-04 10:45:20 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758