We are aware of this problem, I'm also in favor of getting this fixed, but it's likely not an easy fix. Ideally kubevirt - or HCO - would detect that we delete thte namespace and doa proper cleanup.
This is probably tricky to solve, but still a convenient thing.
I think we would need the virt-operator to do 'oc delete apiservices v1alpha3.subresources.kubevirt.io' when the virt cr is removed.
Marc, the workaround that is done is to call: oc delete apiservices v1alpha3.subresources.kubevirt.io This sounds like the virt-operator does not remove the apiservice registration upon removal, is this the case?
Lev, can you please describe and point to the logic that removes KubeVirt?
(In reply to Fabian Deutsch from comment #10) > Lev, can you please describe and point to the logic that removes KubeVirt? We first delete the relevant KubeVirt CRs, which should cause the KubeVirt to stop the relevant services - this is done by the HCO operator itself once the HCO CR is deleted. Then we delete the KubeVirt (and other) operators from the kubevirt-hyperconverged ns, (the ones that were manually started). Basically the current QA flow is the one from the Comment #2.
Alright, it sounds like we should tkae care to remove all components if our CR is getting removed.
@Pan, I would add a reference to that Bug and also duplicate it in the Known issues section, again with a reference to the bug.
@Lev, you said one shouldn't remove CSV manually, but in the console, there is 'Edit and Delete Cluster Service Version' buttons in Catalog -> Installed Operators -> 3 dots menu next to KubeVirt HyperConverged Cluster Operator. If one hits that Delete button, the CSV will be removed. Should that button exist at all, then?
(In reply to Irina Gulina from comment #19) > @Pan, I would add a reference to that Bug and also duplicate it in the Known > issues section, again with a reference to the bug. In theory you can delete any resource and CSV is not an exception, not sure if we should start adding "foolproof"protections to the UI.
Pan, thanks. It looks fine.
*** This bug has been marked as a duplicate of bug 1685911 ***
*** Bug 1685911 has been marked as a duplicate of this bug. ***
https://github.com/kubevirt/kubevirt/pull/3006 is posted. I am not sure if it will solve the issue for HCO as a whole, because other CNV components (like CDI) could still block the delete. Michael I guess you can immediately tell if you delete the apiserver registration when CDI gets uninstalled.
Interesting question, Roman. CDI mostly depends on owner references to clean up when CDI is removed. Hence, the CDI CR owns the ApiService. So it should get deleted when the CDI CR is deleted. But it's not. At some point after the ApiService is created it appears to get updated and the owner reference is removed. Along with other spec data. Not sure where it's happening but I'm pretty confident it's not caused by a bug in CDI. The end result is that the apiservice does not get deleted along with the rest of CDI. Strangely, deleting with kubectl does not delete it either. Not sure what is going on. Must be the k8s bug Stu mentioned earlier? Nevertheless, in my testing, this has not kept me from deleting the CDI install namespace. One more note, CDI also puts owner references on mutating/valiating webhooks and those get cleaned up just as expected.
Created attachment 1655693 [details] virt-api marked as ServiceNotFound Update: in 2.2, after deleting HCO CR, virt-api service is marked as ServiceNotFound, but not deleted. Yaml is still available after some time passes.
Roman, What's the state of this?
This should be addressed in 2.3 with https://github.com/kubevirt/kubevirt/pull/3006.
verify with build: $oc version Client Version: 4.4.0-0.nightly-2020-02-17-022408 Server Version: 4.4.0-0.nightly-2020-02-22-102956 Kubernetes Version: v1.17.1 step: 1.deploy HCO $ oc get hco NAME AGE kubevirt-hyperconverged 42m 2.delete HCO $ oc delete hco kubevirt-hyperconverged -n openshift-cnv hyperconverged.hco.kubevirt.io "kubevirt-hyperconverged" deleted $ oc get apiservices v1alpha3.subresources.kubevirt.io -n openshift-cnv Error from server (NotFound): apiservices.apiregistration.k8s.io "v1alpha3.subresources.kubevirt.io" not found $ oc delete sub hco-operatorhub -n openshift-cnv subscription.operators.coreos.com "hco-operatorhub" deleted $ oc delete catsrc rh-verified-operators -n openshift-marketplace catalogsource.operators.coreos.com "rh-verified-operators" deleted $ oc delete csv kubevirt-hyperconverged-operator.v2.3.0 -n openshift-cnv clusterserviceversion.operators.coreos.com "kubevirt-hyperconverged-operator.v2.3.0" deleted $ oc delete og openshift-cnv-group -n openshift-cnv operatorgroup.operators.coreos.com "openshift-cnv-group" deleted $ oc delete project openshift-cnv project.project.openshift.io "openshift-cnv" deleted $ oc describe project openshift-cnv Error from server (NotFound): namespaces "openshift-cnv" not foun the project can be deleted move to verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2020:2011