Bug 2108634

Summary:	Submariner gateway node does not get deleted with subctl cloud cleanup command
Product:	Red Hat Advanced Cluster Management for Kubernetes	Reporter:	Rahul Lepakshi <rlepaksh>
Component:	Submariner	Assignee:	Aswin Suryanarayanan <asuryana>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Noam Manos <nmanos>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	rhacm-2.5.z	CC:	maafried, ngangadh, nyechiel
Target Milestone:	---	Flags:	bot-tracker-sync: rhacm-2.7+ nyechiel: rhacm-2.7.z+
Target Release:	rhacm-2.7
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	2.7	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2023-01-31 21:49:34 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Comment 1 Aswin Suryanarayanan 2022-07-19 15:44:02 UTC

@rlepaksh Was the machineset deleted from openshift-machine-api namespace(the machineset that has submariner in its name)? If not you can delete that manually before running prepare again while we explore the issue.

Comment 3 Nir Yechiel 2022-09-19 09:05:32 UTC

A Submariner fix is under review: https://github.com/submariner-io/cloud-prepare/pull/383

Comment 4 Noam Manos 2022-11-17 22:43:07 UTC

This is the output on ACM 2.6.2 with Submariner 0.13.1:

$▶ subctl cloud cleanup rhos --infra-id osp-nmanos-b2-8w9fj --region regionOne --project-id "multi-cluster-networking"
 ✓ Retrieving RHOS credentials from your RHOS configuration 
 ✓ Removing the Submariner gateway configuration from nodes  
 ✓ Deleting the gateway instance "osp-nmanos-b2-8w9fj-submariner-gw-0-kdwdg" 
 ✓ Successfully deleted the instance
 ✓ Successfully removed the Submariner gateway configuration from the nodes
⢎⡱ Deleting the Submariner gateway security group Failed to cleanup RHOS cloud: error deleting the Submariner gateway security group: error deleting the security group "osp-nmanos-b2-8w9fj-submariner-gw-sg": Bad request with: [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13774/v2.1/os-security-groups/ad9b5501-a4a5-4779-99d6-8c2c4fab1eac], error message: {"badRequest": {"code": 400, "message": "Security Group ad9b5501-a4a5-4779-99d6-8c2c4fab1eac in use.\nNeutron server returns request_ids: ['req-935df88d-7836-45e8-aa8d-81be6c3c83eb']"}}


I will reattempt with ACM 2.7.0 with Submariner 0.14.0, and update on results.

Comment 5 Noam Manos 2023-01-16 18:51:04 UTC

Verified on ACM 2.7.0 with Subctl 0.14.1:

# Before cloud cleanup - Submariner Gateway node exists:

$ oc  get nodes -o wide
NAME                                        STATUS   ROLES    AGE     VERSION           INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                                                        KERNEL-VERSION                 CONTAINER-RUNTIME
osp-nmanos-b2-f6tqd-master-0                Ready    master   6d16h   v1.24.6+5658434   10.167.1.144   <none>        Red Hat Enterprise Linux CoreOS 411.86.202212072103-0 (Ootpa)   4.18.0-372.36.1.el8_6.x86_64   cri-o://1.24.3-7.rhaos4.11.gitae53655.el8
osp-nmanos-b2-f6tqd-master-1                Ready    master   6d16h   v1.24.6+5658434   10.167.2.241   <none>        Red Hat Enterprise Linux CoreOS 411.86.202212072103-0 (Ootpa)   4.18.0-372.36.1.el8_6.x86_64   cri-o://1.24.3-7.rhaos4.11.gitae53655.el8
osp-nmanos-b2-f6tqd-master-2                Ready    master   6d16h   v1.24.6+5658434   10.167.3.17    <none>        Red Hat Enterprise Linux CoreOS 411.86.202212072103-0 (Ootpa)   4.18.0-372.36.1.el8_6.x86_64   cri-o://1.24.3-7.rhaos4.11.gitae53655.el8
osp-nmanos-b2-f6tqd-submariner-gw-0-wfdg9   Ready    worker   6d2h    v1.24.6+5658434   10.167.2.253   <none>        Red Hat Enterprise Linux CoreOS 411.86.202212072103-0 (Ootpa)   4.18.0-372.36.1.el8_6.x86_64   cri-o://1.24.3-7.rhaos4.11.gitae53655.el8
osp-nmanos-b2-f6tqd-worker-0-9xrwf          Ready    worker   6d16h   v1.24.6+5658434   10.167.3.246   <none>        Red Hat Enterprise Linux CoreOS 411.86.202212072103-0 (Ootpa)   4.18.0-372.36.1.el8_6.x86_64   cri-o://1.24.3-7.rhaos4.11.gitae53655.el8
osp-nmanos-b2-f6tqd-worker-0-fwbmf          Ready    worker   6d16h   v1.24.6+5658434   10.167.2.68    <none>        Red Hat Enterprise Linux CoreOS 411.86.202212072103-0 (Ootpa)   4.18.0-372.36.1.el8_6.x86_64   cri-o://1.24.3-7.rhaos4.11.gitae53655.el8
osp-nmanos-b2-f6tqd-worker-0-lnb2c          Ready    worker   6d16h   v1.24.6+5658434   10.167.1.122   <none>        Red Hat Enterprise Linux CoreOS 411.86.202212072103-0 (Ootpa)   4.18.0-372.36.1.el8_6.x86_64   cri-o://1.24.3-7.rhaos4.11.gitae53655.el8

# Running Cloud Cleanup:

$ subctl cloud cleanup rhos --infra-id $ocp_infra_id --region regionOne --project-id "multi-cluster-networking" 
 ✓ Retrieving RHOS credentials from your RHOS configuration 
 ✓ Removing the Submariner gateway security group rules from node "osp-nmanos-b2-f6tqd-submariner-gw-0" 
 ✓ Successfully removed security group rules from node "osp-nmanos-b2-f6tqd-submariner-gw-0"
 ✓ Deleting the gateway instance "osp-nmanos-b2-f6tqd-submariner-gw-0" 
 ✓ Successfully deleted the instance
 ✓ Successfully removed the Submariner gateway label from the nodes
 ✓ Deleting the Submariner gateway security group 
 ✓ Successfully deleted the Submariner gateway security group
⠈⡱ Revoking intra-cluster communication permissions


$ oc  get nodes -o wide
NAME                                 STATUS   ROLES    AGE     VERSION           INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                                                        KERNEL-VERSION                 CONTAINER-RUNTIME
osp-nmanos-b2-f6tqd-master-0         Ready    master   6d16h   v1.24.6+5658434   10.167.1.144   <none>        Red Hat Enterprise Linux CoreOS 411.86.202212072103-0 (Ootpa)   4.18.0-372.36.1.el8_6.x86_64   cri-o://1.24.3-7.rhaos4.11.gitae53655.el8
osp-nmanos-b2-f6tqd-master-1         Ready    master   6d16h   v1.24.6+5658434   10.167.2.241   <none>        Red Hat Enterprise Linux CoreOS 411.86.202212072103-0 (Ootpa)   4.18.0-372.36.1.el8_6.x86_64   cri-o://1.24.3-7.rhaos4.11.gitae53655.el8
osp-nmanos-b2-f6tqd-master-2         Ready    master   6d16h   v1.24.6+5658434   10.167.3.17    <none>        Red Hat Enterprise Linux CoreOS 411.86.202212072103-0 (Ootpa)   4.18.0-372.36.1.el8_6.x86_64   cri-o://1.24.3-7.rhaos4.11.gitae53655.el8
osp-nmanos-b2-f6tqd-worker-0-9xrwf   Ready    worker   6d16h   v1.24.6+5658434   10.167.3.246   <none>        Red Hat Enterprise Linux CoreOS 411.86.202212072103-0 (Ootpa)   4.18.0-372.36.1.el8_6.x86_64   cri-o://1.24.3-7.rhaos4.11.gitae53655.el8
osp-nmanos-b2-f6tqd-worker-0-fwbmf   Ready    worker   6d16h   v1.24.6+5658434   10.167.2.68    <none>        Red Hat Enterprise Linux CoreOS 411.86.202212072103-0 (Ootpa)   4.18.0-372.36.1.el8_6.x86_64   cri-o://1.24.3-7.rhaos4.11.gitae53655.el8
osp-nmanos-b2-f6tqd-worker-0-lnb2c   Ready    worker   6d16h   v1.24.6+5658434   10.167.1.122   <none>        Red Hat Enterprise Linux CoreOS 411.86.202212072103-0 (Ootpa)   4.18.0-372.36.1.el8_6.x86_64   cri-o://1.24.3-7.rhaos4.11.gitae53655.el8