Description of problem: Now, after https://bugzilla.redhat.com/show_bug.cgi?id=2026336 on an SNO cluster we have only one replica of virt-api and virt-controller (although we still have 2 virt-operators due to https://github.com/operator-framework/operator-lifecycle-manager/issues/2453 ). When the user will delete HCO CR, HCO will try to delete kubert CR and it succeed but on SNO the kubevirt apiservices are still there as leftovers: $ oc get apiservices | grep virt-api v1.subresources.kubevirt.io kubevirt-hyperconverged/virt-api False (MissingEndpoints) 42m v1alpha3.subresources.kubevirt.io kubevirt-hyperconverged/virt-api False (MissingEndpoints) 42m and this prevents the namespace from being successfully cleaned up, when the user will try to clean up the namespace it will get stuck with: message: 'Discovery failed for some groups, 2 failing: unable to retrieve the complete list of server APIs: subresources.kubevirt.io/v1: the server is currently unable to handle the request, subresources.kubevirt.io/v1alpha3: the server is currently unable to handle the request' reason: DiscoveryFailed status: "True" type: NamespaceDeletionDiscoveryFailure Version-Release number of selected component (if applicable): 4.10 How reproducible: pretty often but not 100% systematic Steps to Reproduce: 1. deploy CNV on SNO 2. remove the HCO CR 3. check for kubevirt apiservices leftovers Actual results: v1.subresources.kubevirt.io and v1alpha3.subresources.kubevirt.io apiservices are still there as leftovers. When the user will try to remove the namespace, it will become stuck with: message: 'Discovery failed for some groups, 2 failing: unable to retrieve the complete list of server APIs: subresources.kubevirt.io/v1: the server is currently unable to handle the request, subresources.kubevirt.io/v1alpha3: the server is currently unable to handle the request' reason: DiscoveryFailed status: "True" type: NamespaceDeletionDiscoveryFailure Expected results: kubevirt-operator successfully removes all its managed resources before removing the finalizer from its CR. The product can be cleanly removed. Additional info: it happens only on SNO with a single instance of virt-api and virt-controller, with 2 instances each everything is smooth.
Created attachment 1846111 [details] kubevirt oeprator logs 1/2
Created attachment 1846112 [details] kubevirt operator logs 2/2
Just for the record, with SNO setup, we currently see this issue, 1) If we decide to have 1 Replica of virt-api and virt-controller , we cannot cleanly remove the product ( this bug ) 2) If we decide to have 2 Replicas of virt-api and virt-controller , we cannot install SR-IOV Operator successfully. https://bugzilla.redhat.com/show_bug.cgi?id=2027420 ( [SNO] SR-IOV operator fails to install after CNV is installed )
We are now able to successfully clean up CNV on SNO without traces of virt-api, apiservices. + oc delete hyperconvergeds --all-namespaces --all --ignore-not-found hyperconverged.hco.kubevirt.io "kubevirt-hyperconverged" deleted ]$ oc get apiservices -n openshift-cnv | grep virt- [kbidarka@localhost cnv]$ openshift-cnv got deleted successfully. + oc delete namespace openshift-cnv --ignore-not-found namespace "openshift-cnv" deleted
Tested with container-native-virtualization/virt-operator/images/v4.10.0-164
oc delete namespace openshift-cnv --ignore-not-found namespace "openshift-cnv" deleted [kbidarka@localhost cnv]$ oc get apiservices -A | grep virt-api [kbidarka@localhost cnv]$ oc projects | grep openshift-cnv [kbidarka@localhost cnv]$ Tested with: HCO Version: v4.10.0-605 Virt-operator: v4.10.0-197 We are now able to successfully clean up CNV on SNO without traces of virt-api, apiservices.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Virtualization 4.10.0 Images security and bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0947