Bug 1833786

Summary:	kubevirt hyperconverged-cluster-operator deploy_marketplace.sh fails in disconnected cluster
Product:	Container Native Virtualization (CNV)	Reporter:	Asher Shoshan <ashoshan>
Component:	Installation	Assignee:	Oren Cohen <ocohen>
Status:	CLOSED ERRATA	QA Contact:	Asher Shoshan <ashoshan>
Severity:	high	Docs Contact:
Priority:	high
Version:	2.4.0	CC:	cnv-qe-bugs, fdeutsch, ncredi, ocohen, stirabos
Target Milestone:	---
Target Release:	2.4.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	https://gitlab.cee.redhat.com/contra/cnv-qe-automation/-/tree/master	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-07-28 19:10:05 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Asher Shoshan 2020-05-10 15:49:20 UTC

Description of problem:
deploy CNV (any version) via https://github.com/kubevirt/hyperconverged-cluster-operator/blob/master/deploy/deploy_marketplace.sh failing in a disconnected environment


Version-Release number of selected component (if applicable):
2.4

How reproducible:


Steps to Reproduce:
1. Deploy disconnected ocp cluster 
2. Deploy CNV via marketplace.sh
3.

Actual results:
script creates operatorSource, which is failing due to no internet connection to quay.io


Expected results:
Not to create an operatorSource, and proceed with existing catalogSource


Additional info:

Comment 1 Nelly Credi 2020-05-13 12:39:09 UTC

@Asher, can you please check if this happens with the new implementation (kustomized)?
If so, we will close this bug

Comment 2 Simone Tiraboschi 2020-05-13 13:07:48 UTC

double checking the code:
https://github.com/kubevirt/hyperconverged-cluster-operator/blob/master/deploy/deploy_marketplace.sh#L96

deploy_marketplace.sh is going to try creating an operatorSource with a specified name (via APP_REGISTRY variable) only if it not existing.
If you pass APP_REGISTRY=redhat-operators it will simply skip and consume the existing operatorSource

The issue was just on the catalogSourceConfig but now we removed it as for https://github.com/kubevirt/hyperconverged-cluster-operator/pull/553 so I think we can move this to modified.

Comment 3 Asher Shoshan 2020-05-13 14:16:47 UTC

(In reply to Nelly Credi from comment #1)
> @Asher, can you please check if this happens with the new implementation
> (kustomized)?
> If so, we will close this bug

Is this (Kustomize) going to be the formal way of deployment?(In reply to Simone Tiraboschi from comment #2)
> double checking the code:
> https://github.com/kubevirt/hyperconverged-cluster-operator/blob/master/
> deploy/deploy_marketplace.sh#L96
> 
> deploy_marketplace.sh is going to try creating an operatorSource with a
> specified name (via APP_REGISTRY variable) only if it not existing.
> If you pass APP_REGISTRY=redhat-operators it will simply skip and consume
> the existing operatorSource
> 
> The issue was just on the catalogSourceConfig but now we removed it as for
> https://github.com/kubevirt/hyperconverged-cluster-operator/pull/553 so I
> think we can move this to modified.

Simone,
In a disconnected cluster -  there is no operatorSource --> so your scripts tries to create it and failing (no connection to quay.io, etc)

Comment 6 Simone Tiraboschi 2020-05-13 16:23:28 UTC

(In reply to Asher Shoshan from comment #3)
> Simone,
> In a disconnected cluster -  there is no operatorSource --> so your scripts
> tries to create it and failing (no connection to quay.io, etc)

did you already tried also with
https://github.com/kubevirt/hyperconverged-cluster-operator/blob/master/deploy/deploy_imageregistry.sh

pointing to the registry image you mirrored?

Comment 7 Asher Shoshan 2020-05-13 17:07:57 UTC

(In reply to Simone Tiraboschi from comment #6)
> (In reply to Asher Shoshan from comment #3)
> > Simone,
> > In a disconnected cluster -  there is no operatorSource --> so your scripts
> > tries to create it and failing (no connection to quay.io, etc)
> 
> did you already tried also with
> https://github.com/kubevirt/hyperconverged-cluster-operator/blob/master/
> deploy/deploy_imageregistry.sh
> 
> pointing to the registry image you mirrored?

But again, this script creates catalogSource, which overrides the existing catalogSource (image pointing to local-registry)..
This will pass, but pod will not come up (no internet connection..), and will mess up the disconnected catalogSource

Comment 11 Simone Tiraboschi 2020-06-09 14:28:50 UTC

Oren, can you please try reproducing this?

Comment 13 Simone Tiraboschi 2020-06-09 16:17:05 UTC

According to the step 4 in the official procedure:
https://docs.openshift.com/container-platform/4.4/operators/olm-restricted-networks.html#olm-restricted-networks-operatorhub_olm-restricted-networks

 Create a CatalogSource object that references your catalog image.
  
  Modify the following to your specifications and save it as a catalogsource.yaml file:
  
  
  apiVersion: operators.coreos.com/v1alpha1
  kind: CatalogSource
  metadata:
    name: my-operator-catalog
    namespace: openshift-marketplace
  spec:
    sourceType: grpc
    image: <registry_host_name>:<port>/olm/redhat-operators:v1 
    displayName: My Operator Catalog
    publisher: grpc
  Specify your custom Operator catalog image.
  Use the file to create the CatalogSource object:
  
  
  $ oc create -f catalogsource.yaml

which is exactly what Oren is suggesting setting HCO_REGISTRY_IMAGE with the kustomize based deployment or using deploy_imageregistry.sh instead of deploy_marketplace.sh

Am I missing the real issue?

Comment 17 Oren Cohen 2020-07-13 08:16:02 UTC

MR has been merged to master branch.

Comment 18 Asher Shoshan 2020-07-21 10:42:11 UTC

deploy_kustomize.sh Failing :

13:29:23 ++ dirname cnv-qe-automation/cnv/deploy_kustomize.sh
13:29:23 + SCRIPT_DIR=/root/workspace/deploy-ocp-4.5-cnv-2.4-on-BM/cnv-qe-automation/cnv
13:29:23 + OC_TOOL=oc
13:29:23 + CNV_VERSION=2.4
13:29:23 + CNV_SOURCE=brew
13:29:23 + HCO_REGISTRY_IMAGE=
13:29:23 + CNV_SOURCES=(brew stage osbs production)
13:29:23 + CNV_VERSIONS=(2.4)
13:29:23 + SOURCES_MAP=(["brew"]="rh-verified-operators" ["stage"]="redhat-operators-stage" ["osbs"]="rh-osbs-operators" ["production"]="redhat-operators")
13:29:23 + declare -A SOURCES_MAP
13:29:23 + main
13:29:23 + fetch_upstream_bases
13:29:23 ++ mktemp -d
13:29:23 + TMPDIR=/tmp/tmp.wJec47evni
13:29:23 + trap 'rm -rf /tmp/tmp.wJec47evni' EXIT
13:29:23 + cd /tmp/tmp.wJec47evni
13:29:23 + curl -L https://api.github.com/repos/kubevirt/hyperconverged-cluster-operator/tarball/master
13:29:23 + tar --strip-components=1 -xvzf - 'kubevirt-hyperconverged-cluster-operator-*/deploy/kustomize'
13:29:23   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
13:29:23                                  Dload  Upload   Total   Spent    Left  Speed
13:29:23 
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
13:29:23 kubevirt-hyperconverged-cluster-operator-0878c67/deploy/kustomize/
13:29:23 kubevirt-hyperconverged-cluster-operator-0878c67/deploy/kustomize/README.md
13:29:23 kubevirt-hyperconverged-cluster-operator-0878c67/deploy/kustomize/base/
13:29:23 kubevirt-hyperconverged-cluster-operator-0878c67/deploy/kustomize/base/hco_cr.yaml
13:29:23 kubevirt-hyperconverged-cluster-operator-0878c67/deploy/kustomize/base/kustomization.yaml
13:29:23 kubevirt-hyperconverged-cluster-operator-0878c67/deploy/kustomize/base/operator_group.yaml
13:29:23 kubevirt-hyperconverged-cluster-operator-0878c67/deploy/kustomize/base/subscription.yaml
13:29:23 kubevirt-hyperconverged-cluster-operator-0878c67/deploy/kustomize/deploy_kustomize.sh
13:29:23 kubevirt-hyperconverged-cluster-operator-0878c67/deploy/kustomize/image_registry/
13:29:23 kubevirt-hyperconverged-cluster-operator-0878c67/deploy/kustomize/image_registry/catalog_source.yaml
13:29:23 kubevirt-hyperconverged-cluster-operator-0878c67/deploy/kustomize/image_registry/kustomization.yaml
13:29:23 kubevirt-hyperconverged-cluster-operator-0878c67/deploy/kustomize/kvm_emulation/
13:29:23 kubevirt-hyperconverged-cluster-operator-0878c67/deploy/kustomize/kvm_emulation/kustomization.yaml
13:29:23 kubevirt-hyperconverged-cluster-operator-0878c67/deploy/kustomize/kvm_emulation/subscription.patch.yaml
13:29:23 kubevirt-hyperconverged-cluster-operator-0878c67/deploy/kustomize/marketplace/
13:29:23 kubevirt-hyperconverged-cluster-operator-0878c67/deploy/kustomize/marketplace/kustomization.yaml
13:29:23 kubevirt-hyperconverged-cluster-operator-0878c67/deploy/kustomize/marketplace/operator_source.yaml
13:29:23 kubevirt-hyperconverged-cluster-operator-0878c67/deploy/kustomize/namespace.yaml
13:29:23 kubevirt-hyperconverged-cluster-operator-0878c67/deploy/kustomize/private_repo/
13:29:23 kubevirt-hyperconverged-cluster-operator-0878c67/deploy/kustomize/private_repo/kustomization.yaml
13:29:23 kubevirt-hyperconverged-cluster-operator-0878c67/deploy/kustomize/private_repo/operator_source.patch.yaml
13:29:24 
100 2337k    0 2337k    0     0  3915k      0 --:--:-- --:--:-- --:--:-- 3915k
100 7201k    0 7201k    0     0  4512k      0 --:--:--  0:00:01 --:--:-- 4864k
100 9037k    0 9037k    0     0  4648k      0 --:--:--  0:00:01 --:--:-- 4973k
13:29:24 + cd /root/workspace/deploy-ocp-4.5-cnv-2.4-on-BM/cnv-qe-automation/cnv
13:29:24 + cp -r /tmp/tmp.wJec47evni/deploy/kustomize /root/workspace/deploy-ocp-4.5-cnv-2.4-on-BM/cnv-qe-automation/cnv
13:29:24 + '[' -z ']'
13:29:24 + echo brew stage osbs production
13:29:24 + grep -q -w brew
13:29:24 + echo 2.4
13:29:24 + grep -q -w 2.4
13:29:24 + bash /root/workspace/deploy-ocp-4.5-cnv-2.4-on-BM/cnv-qe-automation/cnv/kustomize/cleanup_leftovers.sh
13:29:24 + export CLUSTER_DIR=/root/ocp/BM-savanna-cluster2
13:29:24 + CLUSTER_DIR=/root/ocp/BM-savanna-cluster2
13:29:24 + export KUBECONFIG=/root/ocp/BM-savanna-cluster2/auth/kubeconfig
13:29:24 + KUBECONFIG=/root/ocp/BM-savanna-cluster2/auth/kubeconfig
13:29:24 + for kind in vmi vm dv
13:29:24 + oc explain vmi
13:29:25 error: Couldn't find resource for "kubevirt.io/v1alpha3, Kind=VirtualMachineInstance"
13:29:25 + continue
13:29:25 + for kind in vmi vm dv
13:29:25 + oc explain vm
13:29:25 error: Couldn't find resource for "kubevirt.io/v1alpha3, Kind=VirtualMachine"
13:29:25 + continue
13:29:25 + for kind in vmi vm dv
13:29:25 + oc explain dv
13:29:26 error: Couldn't find resource for "cdi.kubevirt.io/v1alpha1, Kind=DataVolume"
13:29:26 + continue
13:29:26 + oc delete --ignore-not-found=true storageclass.storage.k8s.io/hostpath-provisioner
13:29:26 + oc delete hostpathprovisioner.hostpathprovisioner.kubevirt.io/hostpath-provisioner
13:29:26 Error from server (NotFound): hostpathprovisioners.hostpathprovisioner.kubevirt.io "hostpath-provisioner" not found
13:29:26 + true
13:29:26 ++ oc get catsrc rh-verified-operators -n openshift-marketplace -o=custom-columns=:metadata.name
13:29:26 + CAT_SRC='
13:29:26 rh-verified-operators'
13:29:26 + '[' -n '
13:29:26 rh-verified-operators' ']'
13:29:26 ++ oc get opsrc rh-verified-operators -n openshift-marketplace -o=custom-columns=:metadata.name
13:29:26 Error from server (NotFound): operatorsources.operators.coreos.com "rh-verified-operators" not found
13:29:26 + OP_SRC=
13:29:26 + true
13:29:26 + oc delete -k /root/workspace/deploy-ocp-4.5-cnv-2.4-on-BM/cnv-qe-automation/cnv/kustomize/brew
13:29:27 secret "quay-registry-kubevirt-hyperconverged" deleted
13:29:27 Error from server (NotFound): error when deleting "/root/workspace/deploy-ocp-4.5-cnv-2.4-on-BM/cnv-qe-automation/cnv/kustomize/brew": operatorsources.operators.coreos.com "kubevirt-hyperconverged" not found
13:29:27 + true
13:29:27 + sed -i 's/- namespace/#- namespace/' /root/workspace/deploy-ocp-4.5-cnv-2.4-on-BM/cnv-qe-automation/cnv/kustomize/deploy_hco_2.4/kustomization.yaml
13:29:27 + oc delete -k /root/workspace/deploy-ocp-4.5-cnv-2.4-on-BM/cnv-qe-automation/cnv/kustomize/deploy_hco_2.4
13:29:27 operatorgroup.operators.coreos.com "kubevirt-hyperconverged-group" deleted
13:29:27 Error from server (NotFound): error when deleting "/root/workspace/deploy-ocp-4.5-cnv-2.4-on-BM/cnv-qe-automation/cnv/kustomize/deploy_hco_2.4": subscriptions.operators.coreos.com "hco-operatorhub" not found
13:29:27 unable to recognize "/root/workspace/deploy-ocp-4.5-cnv-2.4-on-BM/cnv-qe-automation/cnv/kustomize/deploy_hco_2.4": no matches for kind "HyperConverged" in version "hco.kubevirt.io/v1beta1"
13:29:27 + true
13:29:27 + sed -i 's/#- namespace/- namespace/' /root/workspace/deploy-ocp-4.5-cnv-2.4-on-BM/cnv-qe-automation/cnv/kustomize/deploy_hco_2.4/kustomization.yaml
13:29:27 ++ oc get csv -n openshift-cnv --no-headers
13:29:27 ++ awk '{print $1}'
13:29:27 No resources found in openshift-cnv namespace.
13:29:27 + CSV_NAME=
13:29:27 + '[' -n '' ']'
13:29:27 + oc wait pod --for=delete -n openshift-cnv --all --timeout=60s
13:29:27 error: no matching resources found
13:29:27 + true
13:29:27 + echo '[INFO] Delivering HCO content to OLM'
13:29:27 [INFO] Delivering HCO content to OLM
13:29:27 + '[' -z ']'
13:29:27 + '[' -z ']'
13:29:27 + '[' -n '
13:29:27 rh-verified-operators' ']'
13:29:27 + echo '[INFO]: Skipping creating an OperatorSource in a disconnected cluster.'
13:29:27 [INFO]: Skipping creating an OperatorSource in a disconnected cluster.
13:29:27 + sed -E -i 's|(source: ).*|\1
13:29:27 rh-verified-operators|' /root/workspace/deploy-ocp-4.5-cnv-2.4-on-BM/cnv-qe-automation/cnv/kustomize/deploy_hco_2.4/subscription.patch.yaml
13:29:27 sed: -e expression #1, char 17: unterminated `s' command
13:29:27 + rm -rf /tmp/tmp.wJec47evni

Comment 19 Oren Cohen 2020-07-21 11:19:50 UTC

Hi Asher,
Have you checked out from master branch of:
https://gitlab.cee.redhat.com/contra/cnv-qe-automation/-/blob/master/cnv/deploy_kustomize.sh
?

Because the master branch contains manifests for 2.5, and for CNV 2.4 you should checkout cnv-2.4-ocp-4.5 branch.

Comment 20 Asher Shoshan 2020-07-21 11:33:52 UTC

I checked out 2.4   (my branch based on this 2.4)  -  see also in log it's taking 2.4

Comment 21 Oren Cohen 2020-07-21 14:08:08 UTC

Correct, the catalog source name was extracted erroneously with a carriage return, thus the error in `sed`.
I submitted an MR which fixes it.
https://gitlab.cee.redhat.com/contra/cnv-qe-automation/-/merge_requests/331

Comment 24 errata-xmlrpc 2020-07-28 19:10:05 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:3194