@rlepaksh Was the machineset deleted from openshift-machine-api namespace(the machineset that has submariner in its name)? If not you can delete that manually before running prepare again while we explore the issue.
A Submariner fix is under review: https://github.com/submariner-io/cloud-prepare/pull/383
This is the output on ACM 2.6.2 with Submariner 0.13.1: $▶ subctl cloud cleanup rhos --infra-id osp-nmanos-b2-8w9fj --region regionOne --project-id "multi-cluster-networking" ✓ Retrieving RHOS credentials from your RHOS configuration ✓ Removing the Submariner gateway configuration from nodes ✓ Deleting the gateway instance "osp-nmanos-b2-8w9fj-submariner-gw-0-kdwdg" ✓ Successfully deleted the instance ✓ Successfully removed the Submariner gateway configuration from the nodes ⢎⡱ Deleting the Submariner gateway security group Failed to cleanup RHOS cloud: error deleting the Submariner gateway security group: error deleting the security group "osp-nmanos-b2-8w9fj-submariner-gw-sg": Bad request with: [DELETE https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13774/v2.1/os-security-groups/ad9b5501-a4a5-4779-99d6-8c2c4fab1eac], error message: {"badRequest": {"code": 400, "message": "Security Group ad9b5501-a4a5-4779-99d6-8c2c4fab1eac in use.\nNeutron server returns request_ids: ['req-935df88d-7836-45e8-aa8d-81be6c3c83eb']"}} I will reattempt with ACM 2.7.0 with Submariner 0.14.0, and update on results.
Verified on ACM 2.7.0 with Subctl 0.14.1: # Before cloud cleanup - Submariner Gateway node exists: $ oc get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME osp-nmanos-b2-f6tqd-master-0 Ready master 6d16h v1.24.6+5658434 10.167.1.144 <none> Red Hat Enterprise Linux CoreOS 411.86.202212072103-0 (Ootpa) 4.18.0-372.36.1.el8_6.x86_64 cri-o://1.24.3-7.rhaos4.11.gitae53655.el8 osp-nmanos-b2-f6tqd-master-1 Ready master 6d16h v1.24.6+5658434 10.167.2.241 <none> Red Hat Enterprise Linux CoreOS 411.86.202212072103-0 (Ootpa) 4.18.0-372.36.1.el8_6.x86_64 cri-o://1.24.3-7.rhaos4.11.gitae53655.el8 osp-nmanos-b2-f6tqd-master-2 Ready master 6d16h v1.24.6+5658434 10.167.3.17 <none> Red Hat Enterprise Linux CoreOS 411.86.202212072103-0 (Ootpa) 4.18.0-372.36.1.el8_6.x86_64 cri-o://1.24.3-7.rhaos4.11.gitae53655.el8 osp-nmanos-b2-f6tqd-submariner-gw-0-wfdg9 Ready worker 6d2h v1.24.6+5658434 10.167.2.253 <none> Red Hat Enterprise Linux CoreOS 411.86.202212072103-0 (Ootpa) 4.18.0-372.36.1.el8_6.x86_64 cri-o://1.24.3-7.rhaos4.11.gitae53655.el8 osp-nmanos-b2-f6tqd-worker-0-9xrwf Ready worker 6d16h v1.24.6+5658434 10.167.3.246 <none> Red Hat Enterprise Linux CoreOS 411.86.202212072103-0 (Ootpa) 4.18.0-372.36.1.el8_6.x86_64 cri-o://1.24.3-7.rhaos4.11.gitae53655.el8 osp-nmanos-b2-f6tqd-worker-0-fwbmf Ready worker 6d16h v1.24.6+5658434 10.167.2.68 <none> Red Hat Enterprise Linux CoreOS 411.86.202212072103-0 (Ootpa) 4.18.0-372.36.1.el8_6.x86_64 cri-o://1.24.3-7.rhaos4.11.gitae53655.el8 osp-nmanos-b2-f6tqd-worker-0-lnb2c Ready worker 6d16h v1.24.6+5658434 10.167.1.122 <none> Red Hat Enterprise Linux CoreOS 411.86.202212072103-0 (Ootpa) 4.18.0-372.36.1.el8_6.x86_64 cri-o://1.24.3-7.rhaos4.11.gitae53655.el8 # Running Cloud Cleanup: $ subctl cloud cleanup rhos --infra-id $ocp_infra_id --region regionOne --project-id "multi-cluster-networking" ✓ Retrieving RHOS credentials from your RHOS configuration ✓ Removing the Submariner gateway security group rules from node "osp-nmanos-b2-f6tqd-submariner-gw-0" ✓ Successfully removed security group rules from node "osp-nmanos-b2-f6tqd-submariner-gw-0" ✓ Deleting the gateway instance "osp-nmanos-b2-f6tqd-submariner-gw-0" ✓ Successfully deleted the instance ✓ Successfully removed the Submariner gateway label from the nodes ✓ Deleting the Submariner gateway security group ✓ Successfully deleted the Submariner gateway security group ⠈⡱ Revoking intra-cluster communication permissions $ oc get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME osp-nmanos-b2-f6tqd-master-0 Ready master 6d16h v1.24.6+5658434 10.167.1.144 <none> Red Hat Enterprise Linux CoreOS 411.86.202212072103-0 (Ootpa) 4.18.0-372.36.1.el8_6.x86_64 cri-o://1.24.3-7.rhaos4.11.gitae53655.el8 osp-nmanos-b2-f6tqd-master-1 Ready master 6d16h v1.24.6+5658434 10.167.2.241 <none> Red Hat Enterprise Linux CoreOS 411.86.202212072103-0 (Ootpa) 4.18.0-372.36.1.el8_6.x86_64 cri-o://1.24.3-7.rhaos4.11.gitae53655.el8 osp-nmanos-b2-f6tqd-master-2 Ready master 6d16h v1.24.6+5658434 10.167.3.17 <none> Red Hat Enterprise Linux CoreOS 411.86.202212072103-0 (Ootpa) 4.18.0-372.36.1.el8_6.x86_64 cri-o://1.24.3-7.rhaos4.11.gitae53655.el8 osp-nmanos-b2-f6tqd-worker-0-9xrwf Ready worker 6d16h v1.24.6+5658434 10.167.3.246 <none> Red Hat Enterprise Linux CoreOS 411.86.202212072103-0 (Ootpa) 4.18.0-372.36.1.el8_6.x86_64 cri-o://1.24.3-7.rhaos4.11.gitae53655.el8 osp-nmanos-b2-f6tqd-worker-0-fwbmf Ready worker 6d16h v1.24.6+5658434 10.167.2.68 <none> Red Hat Enterprise Linux CoreOS 411.86.202212072103-0 (Ootpa) 4.18.0-372.36.1.el8_6.x86_64 cri-o://1.24.3-7.rhaos4.11.gitae53655.el8 osp-nmanos-b2-f6tqd-worker-0-lnb2c Ready worker 6d16h v1.24.6+5658434 10.167.1.122 <none> Red Hat Enterprise Linux CoreOS 411.86.202212072103-0 (Ootpa) 4.18.0-372.36.1.el8_6.x86_64 cri-o://1.24.3-7.rhaos4.11.gitae53655.el8