Bug 2097381

Summary:	[Submariner] - Fails to increase gateway amount after deployment
Product:	Red Hat Advanced Cluster Management for Kubernetes	Reporter:	Maxim Babushkin <mbabushk>
Component:	Submariner	Assignee:	Mike Kolesnik <mkolesni>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Noam Manos <nmanos>
Severity:	medium	Docs Contact:	Christopher Dawson <cdawson>
Priority:	unspecified
Version:	rhacm-2.5	CC:	dfarrell, ecai, maafried, mkolesni, nyechiel, skitt
Target Milestone:	---	Keywords:	Reopened
Target Release:	rhacm-2.7	Flags:	nyechiel: rhacm-2.7+ nyechiel: rhacm-2.7.z+
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2023-01-31 21:49:34 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Maxim Babushkin 2022-06-15 14:50:32 UTC

**What happened**:
ACM 2.5
Submariner 0.12.1
Deployment platform AWS and GCP

Fails to increase the amount of gateways on clusters after deployment.
After successful deployment of submariner, increase the amount of submariner gateways for the clusters, failed.

On GCP platform, after increasing the amount of gateway nodes on submariner config from the hub, nothing happen.
The SubmarinerConfig state indicates the the number of gateway nodes is not desired number, but nothing happens.
From the gcp console nothing is happening as well. No new gateway vm created.

On AWS platform, the issue is similar to the following BZ - https://bugzilla.redhat.com/show_bug.cgi?id=2093990

No error logs on the hub addon or cluster addon.

**What you expected to happen**:
When increasing the amount of the gateway nodes after deployment, it should reach the desired state with no issues.

**How to reproduce it (as minimally and precisely as possible)**:
1) Deploy an acm hub and two managed clusters.
2) Deploy submariner addon on both clusters with 1 single gateway node.
3) After successful deployment, increase the amount of the gateway by 1 from the UI or cli.

**Anything else we need to know?**:

**Environment**:
- Submariner version (use `subctl version`): 0.12.1
- Kubernetes version (use `kubectl version`): 4.10.18
- Logs attached.

Comment 9 Maxim Babushkin 2022-08-10 12:11:46 UTC

Tested on the latest 2.6 / 0.13.0 build.
The increase is working for the AWS platform. Was fixes by https://github.com/stolostron/submariner-addon/pull/409

But still not works for the GCP platform.

Comment 10 Mike Kolesnik 2022-08-16 09:29:24 UTC

This needs https://github.com/stolostron/submariner-addon/issues/452 fixed in order to be able to analyze & fix it properly.

Comment 11 Daniel Farrell 2022-08-25 12:51:05 UTC

> This needs https://github.com/stolostron/submariner-addon/issues/452 fixed in order to be able to analyze & fix it properly.

This has been merged.

Comment 12 Mike Kolesnik 2022-08-29 13:18:23 UTC

The problem is the operator is consumed from a subscription, and the image there is 0.13.0 IIRC, and theres no convenient way to use a different one.
We should probably get https://github.com/stolostron/submariner-addon/issues/475 in to be able to resume testing (this and other issues).

Comment 13 Maayan Friedman 2022-10-20 13:50:22 UTC

Can you try test with the workaround described here: https://github.com/stolostron/submariner-addon/pull/522 ?

Comment 14 Mike Kolesnik 2022-11-28 10:12:32 UTC

This should be fixed by https://github.com/stolostron/submariner-addon/pull/606 (which would need backports to 2.7 & 2.6 & perhaps 2.5?)

Comment 15 Nir Yechiel 2022-11-28 12:05:39 UTC

2.6 backport: https://github.com/stolostron/submariner-addon/pull/609

Comment 16 Maxim Babushkin 2022-12-07 12:08:41 UTC

Increase of the gateway nodes post deployment has been successfully verified.