Bug 1966513

Summary: Unable to make SSH connection to a Bitbucket server
Product: Red Hat Advanced Cluster Management for Kubernetes Reporter: Riya Banerjee <ribanerj>
Component: App LifecycleAssignee: Roke Jung <rjung>
Status: CLOSED ERRATA QA Contact: Eveline Cai <ecai>
Severity: unspecified Docs Contact: bswope <bswope>
Priority: unspecified    
Version: rhacm-2.2.zCC: mlele, rjung, xiangli
Target Milestone: ---Flags: ming: rhacm-2.2.z+
Target Release: rhacm-2.2.4   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1973595 (view as bug list) Environment:
Last Closed: 2021-06-16 19:28:30 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1973595, 2028196    

Description Riya Banerjee 2021-06-01 10:08:48 UTC 
Description of the problem:
SSH connection to a Bitbucket server fails with below error:-
~~~
2021-05-28T08:06:50.563908948Z I0528 08:06:50.563895       1 gitrepo.go:177] Connecting to Git server via SSH
2021-05-28T08:06:50.563917741Z I0528 08:06:50.563907       1 gitrepo.go:220] Getting public SSH host key for bitbucket.biscrum.com
2021-05-28T08:06:55.606535708Z E0528 08:06:55.606491       1 gitrepo.go:226] failed to get public SSH host key: exit status 1
2021-05-28T08:06:55.606535708Z E0528 08:06:55.606515       1 git_subscriber_item.go:195] exit status 1Unable to clone the git repo ssh://git.com:7999/infpaas/openshift.git
2021-05-28T08:06:55.606535708Z I0528 08:06:55.606522       1 git_subscriber_item.go:198] exit doSubscription: policies/stable-policies-sub
2021-05-28T08:06:55.606535708Z E0528 08:06:55.606528       1 git_subscriber_item.go:162] exit status 1Subscription error.
~~~

Release version: RHACM 2.2

OCP version: 4.7.12

Steps to reproduce:
1. Create the RSA Key Pair
2. Add the public key to your repository on bitbucket server
3. Create a secret and configure channel as per steps in below document:
https://access.redhat.com/documentation/en-us/red_hat_advanced_cluster_management_for_kubernetes/2.2/html/manage_applications/managing-applications#git-SSH-connection

Actual results: 
The multicluster-operators-standalone-subscription pod is throwing this error:
~~~
2021-05-28T08:06:50.563908948Z I0528 08:06:50.563895       1 gitrepo.go:177] Connecting to Git server via SSH
2021-05-28T08:06:50.563917741Z I0528 08:06:50.563907       1 gitrepo.go:220] Getting public SSH host key for bitbucket.biscrum.com
2021-05-28T08:06:55.606535708Z E0528 08:06:55.606491       1 gitrepo.go:226] failed to get public SSH host key: exit status 1
2021-05-28T08:06:55.606535708Z E0528 08:06:55.606515       1 git_subscriber_item.go:195] exit status 1Unable to clone the git repo ssh://git.com:7999/infpaas/openshift.git
2021-05-28T08:06:55.606535708Z I0528 08:06:55.606522       1 git_subscriber_item.go:198] exit doSubscription: policies/stable-policies-sub
2021-05-28T08:06:55.606535708Z E0528 08:06:55.606528       1 git_subscriber_item.go:162] exit status 1Subscription error.
~~~

Expected results: SSH connection is successful without any error.

Additional info:
git clone working fine on the bastion host using same keypair:
~~~
[us-acm@aws-us-bastion ~]$ echo `head -c 100 .ssh/known_hosts`
[bitbucket.biscrum.com]:7999,[148.188.5.28]:7999 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC8sQY4Ca7z2jt
[us-acm@aws-us-bastion ~]$ git clone ssh://git.com:7999/infpaas/openshift.git
Cloning into 'openshift'...
remote: Enumerating objects: 7409, done.
remote: Counting objects: 100% (7409/7409), done.
remote: Compressing objects: 100% (4065/4065), done.
remote: Total 7409 (delta 4616), reused 5355 (delta 3215)
Receiving objects: 100% (7409/7409), 3.90 MiB | 4.54 MiB/s, done.
Resolving deltas: 100% (4616/4616), done.
[us-acm@aws-us-bastion ~]$
~~~
Comment 3 Roke Jung 2021-06-01 13:36:56 UTC 
Replace ssh://git.com:7999/infpaas/openshift.git with git.com:7999/infpaas/openshift.git and try again.
Comment 5 Roke Jung 2021-06-01 16:07:07 UTC 
It looks like the problem is that the controller is trying to run "ssh-keyscan bitbucket.biscrum.com" without the port. Can the customer try to run these two commands on the bastion host and give us the output?

ssh-keyscan bitbucket.biscrum.com

ssh-keyscan bitbucket.biscrum.com:7999
Comment 6 Roke Jung 2021-06-01 18:42:33 UTC 
Also, please check your DNS configuration for bitbucket.biscrum.com VS bitbucket.biscrum.com:7999. If the hostname entry in DNS is bitbucket.biscrum.com:7999, is it possible to update the DNS entry to bitbucket.biscrum.com?
Comment 7 Mike Ng 2021-06-01 18:43:37 UTC 
G2Bsync 852310284 comment 
 rokej Tue, 01 Jun 2021 17:29:09 UTC 
 G2Bsync

Also, please check your DNS configuration for bitbucket.biscrum.com VS bitbucket.biscrum.com:7999. If the hostname entry in DNS is `bitbucket.biscrum.com:7999`, is it possible to change that to `bitbucket.biscrum.com`?
Comment 9 Roke Jung 2021-06-03 13:49:27 UTC 
Does it work if you set the SSH URL to git.com/infpaas/openshift.git? If it still does not work, we can provide a patch until the fix is available in the next maintenance release.
Comment 11 Roke Jung 2021-06-04 15:04:10 UTC 
Sorry, can we try one more time? Set the SSH URL to git.com:infpaas/openshift.git.

Also, is using HTTP also an option?
Comment 13 Roke Jung 2021-06-07 17:07:08 UTC 
Please follow this instruction https://github.com/open-cluster-management/multicloud-operators-subscription/blob/main/docs/patching_subscription_image.md to patch the controller images on your ACM cluster. Use image tag 2.2-59247adbadd8c77e774d60e227b4364cbcd2f160 when you follow the instruction.

Once patched and pods restart with the new image, try the application subscription again using the original SSH URL ssh://git.com:7999/infpaas/openshift.git and provide logs if it still fails. Thank you.
Comment 15 Roke Jung 2021-06-08 12:21:28 UTC 
2.2.4 will be available sometime next week. Did the patch work?
Comment 23 errata-xmlrpc 2021-06-16 19:28:30 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: Red Hat Advanced Cluster Management 2.2.4 security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2461
Comment 25 Roke Jung 2021-06-17 13:31:00 UTC 
Can you please ask the customer for the logs? Thanks.
Comment 26 Roke Jung 2021-06-17 13:31:42 UTC 
Please open a new one. Thanks.