Bug 1757705 - Resources left after cnv uninstall, ns can't be removed, reinstall script will fail
Summary: Resources left after cnv uninstall, ns can't be removed, reinstall script wil...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Container Native Virtualization (CNV)
Classification: Red Hat
Component: Documentation
Version: 2.1.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 2.1.0
Assignee: Andrew Burden
QA Contact: Irina Gulina
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-10-02 08:55 UTC by Irina Gulina
Modified: 2019-11-04 15:04 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-11-04 15:04:58 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
csv copied in all namespaces (418.55 KB, image/png)
2019-10-02 08:55 UTC, Irina Gulina
no flags Details
2 csv left after alpha uninstall (97.65 KB, image/png)
2019-10-02 08:57 UTC, Irina Gulina
no flags Details
4 csv left after 2.1.0 uninstall (124.82 KB, image/png)
2019-10-02 08:59 UTC, Irina Gulina
no flags Details

Description Irina Gulina 2019-10-02 08:55:43 UTC
Created attachment 1621769 [details]
csv copied in all namespaces

Section Number and Name: 
Uninstall docs

Describe the issue: 

With a new feature of OLM for operator groups, when an operator is installed in one ns, its csv copied in all other ns'es. See the screenshot attached.

If then, one uninstalls CNV, not every copy of csv will be removed, see another two screenshots attached. Uninstall of alpha and non-alpha versions leave a different number of csv left in other ns'es. Those can be removed manually, and won't be recreated.

Then, operatorgroup and catalogsource also will stay. It means one can't follow our uninstall docs and remove openshift-cnv ns with those resources left. It also means our installation scripts will fail of one reruns them. ATM, I can't confirm whether reinstall from OperatorHub will fail too, especially if one version is removed and another one to be installed. 

Suggestions for improvement:
Our uninstall docs should mention that resources leftovers may remain, and if so, ns can't be removed, and reinstall also may fail (I will clarify it, when I check that scenario).

Comment 1 Irina Gulina 2019-10-02 08:57:14 UTC
Created attachment 1621770 [details]
2 csv left after alpha uninstall

Comment 2 Irina Gulina 2019-10-02 08:59:41 UTC
Created attachment 1621772 [details]
4 csv left after 2.1.0 uninstall

Comment 3 Irina Gulina 2019-10-02 09:02:04 UTC
After 2.1.0 unistall (via UI, following our uninstall docs):

[root@cnv-qe-03 ~]# oc get og --all-namespaces
NAMESPACE                              NAME                           AGE
local-storage                          local-operator-group           9h
openshift-cnv                          hco-operatorgroup              9h
openshift-monitoring                   openshift-cluster-monitoring   10h
openshift-operator-lifecycle-manager   olm-operators                  10h
openshift-operators                    global-operators               10h

[cnv-qe-jenkins@cnv-executor-igulina-upgrade ~]$ oc get catsrc --all-namespaces
NAMESPACE               NAME                       DISPLAY                    TYPE   PUBLISHER         AGE
openshift-marketplace   certified-operators        Certified Operators        grpc   Red Hat           11h
openshift-marketplace   community-operators        Community Operators        grpc   Red Hat           11h
openshift-marketplace   hco-catalogsource          KubeVirt HyperConverged    grpc   Red Hat           11h
openshift-marketplace   redhat-local-storage-src   redhat stroage operators   grpc   Red Hat Storage   11h
openshift-marketplace   redhat-operators           Red Hat Operators          grpc   Red Hat           99m

Comment 4 Audrey Spaulding 2019-10-02 12:58:39 UTC
Hi Irina,

Is this for the Known Issue section of the release notes?

Comment 5 Irina Gulina 2019-10-02 14:17:51 UTC
Another resource leftover: https://bugzilla.redhat.com/show_bug.cgi?id=1755009

Audrey, not sure about Known Issue section, but it certainly compromise the last step of Uninstall section "Deleting a project using the web console". It won't be possible to delete a project with those resource leftovers. 

Fabian, WDYT about comment #0, please? Is it just documentation or bigger? I believe, Uninstall procedure should remove *everything*, without any leftovers to be removed manually, otherwise it's a bug.

Comment 6 Audrey Spaulding 2019-10-02 14:21:00 UTC
Fabian is on PTO through the end of the week. Maybe Stu Gott can help?

Comment 7 sgott 2019-10-02 15:00:24 UTC
Do these leftover artifacts prevent update, or just a "clean uninstall"? If it's not possible to update to newer versions because of previous CSV's, that's a much bigger problem--because it limits our ability to fix the situation in a future release.

Understood that this affects the ability to re-install if the previous instance wasn't cleanly uninstalled. But in that case, documenting a list of cruft can help the admin fix the situation.

I suggest that for 2.1.0 we do a release note, and target a fix for 2.1.1.

Comment 8 Andrew Burden 2019-10-03 16:34:50 UTC
Working on a release note to cover this for 2.1. Can I please confirm what's remaining after uninstall:
1) `service/kubemacpool-service` in openshift-cnv namespace
2) `hco-operatorgroup` in openshift-cnv namespace
3) `hco-catalogsource` in openshift-marketplace namespace, and
4) `hco-operator` CSV copied in the following namespaces:
openshift-operator-lifecycle-manager
openshift-operators
local-storage
openshift-monitoring

Can any/all of these artefacts be manually deleted? And if so, does that mean CNV could be re-installed?

Comment 9 Irina Gulina 2019-10-07 09:40:59 UTC
regarding 4), I would write smth like: check if there is any "kubevirt-hyperconverged-operator.v2.1.0" csv files left in other namespaces:
oc get csv --all-namespaces | grep kubevirt-hyperconverged-operator, and remove those.

all can be deleted manually yes, 
yes, cnv can bre re-installed after

Comment 10 Andrew Burden 2019-10-08 10:52:01 UTC
Release note written and acked.
Will be included in https://github.com/openshift/openshift-docs/pull/16756


Note You need to log in before you can comment on or make changes to this bug.