Bug 2105885 - ACM 2.5 cannot create known_hosts file when pulling from ssh git repo
Summary: ACM 2.5 cannot create known_hosts file when pulling from ssh git repo
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Advanced Cluster Management for Kubernetes
Classification: Red Hat
Component: App Lifecycle
Version: rhacm-2.5
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: ---
: rhacm-2.5.2
Assignee: Roke Jung
QA Contact: Almen Ng
bswope@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-07-11 05:09 UTC by Mihir Lele
Modified: 2022-10-13 14:37 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-10-03 20:23:57 UTC
Target Upstream Version:
Embargoed:
bot-tracker-sync: rhacm-2.5.z+


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github stolostron backlog issues 24144 0 None None None 2022-07-11 09:50:51 UTC

Description Mihir Lele 2022-07-11 05:09:09 UTC
Description of the problem:

ACM 2.5 cannot create known_hosts file when pulling from ssh git repo

Steps to reproduce:

Create an application from ACM using ssh key 

Additional info:

After installing ACM & configuring a multicluster hub, we do the following steps:

```
cat <<EOF | oc apply -f -
apiVersion: v1
kind: Namespace
metadata:
  name: bit-ros-acm-policies
EOF

cat <<EOFSEC | oc apply -n bit-ros-acm-policies -f -
apiVersion: v1
kind: Secret
metadata:
  name: repo-ros-acm-policies
  namespace: bit-ros-acm-policies
type: Opaque
data:
  sshKey: $(echo -n "${ssh_privatekey}" | base64 -w 0)
EOFSEC

cat <<EOFCHAN | oc apply -n bit-ros-acm-policies -f -
apiVersion: apps.open-cluster-management.io/v1
kind: Channel
metadata:
  annotations:
    apps.open-cluster-management.io/reconcile-rate: high
  name: ros-policies-channel
  namespace: bit-ros-acm-policies
spec:
  secretRef:
    name: repo-ros-acm-policies
  pathname: ssh://git.ch/repo/ros-cluster.git
  type: Git
EOFCHAN

cat <<EOFSUB | oc apply -n bit-ros-acm-policies -f -
apiVersion: apps.open-cluster-management.io/v1
kind: Subscription
metadata:
  annotations:
    apps.open-cluster-management.io/git-branch: master
    apps.open-cluster-management.io/git-path: acm-policies/overlays/${CLUSTER_NAME}
  name: ros-policies-subscription
  namespace: bit-ros-acm-policies
spec:
  channel: bit-ros-acm-policies/ros-policies-channel
  placement:
    local: true
EOFSUB
```

However no policies are synced out of the git-path overlay into ACM.

The Channel has the following status:

```
apiVersion: apps.open-cluster-management.io/v1
kind: Subscription
metadata:
  annotations:
    apps.open-cluster-management.io/git-branch: master
    apps.open-cluster-management.io/git-path: acm-policies/overlays/CLUSTER_NAME
  creationTimestamp: "2022-06-29T11:16:34Z"
  generation: 1
  name: ros-policies-subscription
  namespace: bit-ros-acm-policies
spec:
  channel: bit-ros-acm-policies/ros-policies-channel
  placement:
    local: true
status:
  ansiblejobs: {}
  appstatusReference: kubectl get appsubstatus -n bit-ros-acm-policies ros-policies-subscription
  lastUpdateTime: "2022-06-29T15:17:54Z"
  message: Active
  phase: Subscribed
```

If we check the logs of the pod multicluster-operators-standalone-subscription-6d78d97d4c-2pqz9 I see the following logs:

```
I0629 15:17:48.708681       1 helm_subscriber_item.go:361] Checking to see if the HelmRelease open-cluster-management/volsync-addon-controller-7c984 exists
I0629 15:17:48.710387       1 helm_subscriber_item.go:380] Checking to see if the HelmRelease open-cluster-management/volsync-addon-controller-7c984 status is populated
I0629 15:17:54.424990       1 git_subscriber_item.go:169] Re-try #1: subcribing to the Git repo
I0629 15:17:54.425017       1 git_subscriber_item.go:185] enter doSubscription: bit-ros-acm-policies/ros-policies-subscription
I0629 15:17:54.430958       1 git_subscriber_item.go:203] Subscribing ...ros-policies-subscription
I0629 15:17:54.449881       1 gitrepo.go:210] Connecting to Git server via SSH
I0629 15:17:54.449905       1 gitrepo.go:404] sshhostname =  bitbucket.example.ch
I0629 15:17:54.449911       1 gitrepo.go:405] sshhostport =  
I0629 15:17:54.449916       1 gitrepo.go:407] Getting public SSH host key for bitbucket.example.ch
I0629 15:17:54.520901       1 gitrepo.go:422] SSH host key: bitbucket.example.ch.ch ssh-rsa AAAACensored
E0629 15:17:54.520929       1 gitrepo.go:425] failed to write known_hosts file: open /tmp/bit-ros-acm-policies/ros-policies-subscription/known_hosts: no such file or directory
E0629 15:17:54.520941       1 gitrepo.go:263] Failed to get Git clone options with the primary channel. Trying the secondary channel.
E0629 15:17:54.520947       1 gitrepo.go:176] no secondary channel to try
E0629 15:17:54.520958       1 git_subscriber_item.go:265] failed to build git connection optionsUnable to clone the git repo ssh://git.ch/repo/ros-cluster.git
I0629 15:17:54.520966       1 git_subscriber_item.go:268] exit doSubscription: bit-ros-acm-policies/ros-policies-subscription
E0629 15:17:54.520972       1 git_subscriber_item.go:173] failed to build git connection optionsSubscription error.
```

If we connect to the pod, we can see that the directory is missing, which might be the issue:

```
$ oc rsh multicluster-operators-standalone-subscription-6d78d97d4c-2pqz9
sh-4.4$ ls -l /tmp/
total 0
drwxr-x---. 9 1001 root 202 Jun 29 11:13 hr-charts
drwxr-xr-x. 3 1001 root  55 Jun 29 14:58 open-cluster-management
sh-4.4$ ls -l /tmp/open-cluster-management/
total 0
drwxr-xr-x. 2 1001 root 6 Jun 29 14:58 hive-clusterimagesets-subscription-fast-0
sh-4.4$ ls -l /tmp/hr-charts/              
total 0
drwxr-x---. 3 1001 root 37 Jun 29 11:13 cluster-lifecycle-9ea7f
drwxr-x---. 3 1001 root 37 Jun 29 11:13 console-chart-9718b
drwxr-x---. 3 1001 root 37 Jun 29 11:13 grc-33032
drwxr-x---. 3 1001 root 37 Jun 29 11:13 management-ingress-fdbd5
drwxr-x---. 3 1001 root 37 Jun 29 11:13 policyreport-00422
drwxr-x---. 3 1001 root 37 Jun 29 11:13 search-prod-8684b
drwxr-x---. 3 1001 root 37 Jun 29 11:13 volsync-addon-controller-7c984
```

After creating the directory within the pod:

```
sh-4.4$ mkdir -p /tmp/bit-ros-acm-policies/ros-policies-subscription
```

The git repository can be synced and the policies appear in ACM

```
E0629 15:22:54.803314       1 git_subscriber_item.go:160] failed to build git connection optionsSubscription error.    
                                                                                                                       
##### here we created the directory in /tmp                
                                                                                                                                                                                                                              
I0629 15:23:54.803766       1 git_subscriber_item.go:169] Re-try #1: subcribing to the Git repo                        
I0629 15:23:54.803797       1 git_subscriber_item.go:185] enter doSubscription: bit-ros-acm-policies/ros-policies-subscription
I0629 15:23:54.811902       1 git_subscriber_item.go:203] Subscribing ...ros-policies-subscription
I0629 15:23:54.830170       1 gitrepo.go:210] Connecting to Git server via SSH                                         
I0629 15:23:54.830196       1 gitrepo.go:404] sshhostname =  bitbucket.example.ch
I0629 15:23:54.830202       1 gitrepo.go:405] sshhostport =                                                            
I0629 15:23:54.830207       1 gitrepo.go:407] Getting public SSH host key for bitbucket.example.ch
I0629 15:23:54.895361       1 gitrepo.go:422] SSH host key: bitbucket.example.ch ssh-rsa AAAABCensored
I0629 15:23:54.895511       1 gitrepo.go:461] Using SSH known host keys                                                
E0629 15:23:54.895684       1 gitrepo.go:176] no secondary channel to try                                              
I0629 15:23:54.895700       1 gitrepo.go:291] Cloning ssh://git.ch/repo/ros-cluster.git into /tmp/bit-ros-acm-policies/ros-policies-subscription
I0629 15:23:54.895707       1 gitrepo.go:293] cloneOptions.DestDir = /tmp/bit-ros-acm-policies/ros-policies-subscription
I0629 15:23:54.895711       1 gitrepo.go:294] cloneOptions.Branch = refs/heads/master
I0629 15:23:54.895718       1 gitrepo.go:295] cloneOptions.CommitHash =
I0629 15:23:54.895722       1 gitrepo.go:296] cloneOptions.RevisionTag =
I0629 15:23:54.895726       1 gitrepo.go:297] cloneOptions.CloneDepth = 1
I0629 15:23:55.193210       1 gitrepo.go:330] Successfully cloned the repo and the current branch is master
I0629 15:23:55.193271       1 git_subscriber_item.go:271] Git commit: 01bCensored
```

Comment 1 Mihir Lele 2022-07-11 05:10:17 UTC
Issue will be fixed in 2.5.2  https://github.com/stolostron/multicloud-operators-subscription/pull/780

Comment 2 Roke Jung 2022-07-11 17:26:49 UTC
The fix will be in 2.5.2

Comment 3 Ishmam Amin 2022-09-07 15:18:43 UTC
In 2.5.2 the Application successfully creates host_files and deploys the application using the SSH key. Logs show successful creation and generates a git commit


Note You need to log in before you can comment on or make changes to this bug.