Description of problem: deploy CNV (any version) via https://github.com/kubevirt/hyperconverged-cluster-operator/blob/master/deploy/deploy_marketplace.sh failing in a disconnected environment Version-Release number of selected component (if applicable): 2.4 How reproducible: Steps to Reproduce: 1. Deploy disconnected ocp cluster 2. Deploy CNV via marketplace.sh 3. Actual results: script creates operatorSource, which is failing due to no internet connection to quay.io Expected results: Not to create an operatorSource, and proceed with existing catalogSource Additional info:
@Asher, can you please check if this happens with the new implementation (kustomized)? If so, we will close this bug
double checking the code: https://github.com/kubevirt/hyperconverged-cluster-operator/blob/master/deploy/deploy_marketplace.sh#L96 deploy_marketplace.sh is going to try creating an operatorSource with a specified name (via APP_REGISTRY variable) only if it not existing. If you pass APP_REGISTRY=redhat-operators it will simply skip and consume the existing operatorSource The issue was just on the catalogSourceConfig but now we removed it as for https://github.com/kubevirt/hyperconverged-cluster-operator/pull/553 so I think we can move this to modified.
(In reply to Nelly Credi from comment #1) > @Asher, can you please check if this happens with the new implementation > (kustomized)? > If so, we will close this bug Is this (Kustomize) going to be the formal way of deployment?(In reply to Simone Tiraboschi from comment #2) > double checking the code: > https://github.com/kubevirt/hyperconverged-cluster-operator/blob/master/ > deploy/deploy_marketplace.sh#L96 > > deploy_marketplace.sh is going to try creating an operatorSource with a > specified name (via APP_REGISTRY variable) only if it not existing. > If you pass APP_REGISTRY=redhat-operators it will simply skip and consume > the existing operatorSource > > The issue was just on the catalogSourceConfig but now we removed it as for > https://github.com/kubevirt/hyperconverged-cluster-operator/pull/553 so I > think we can move this to modified. Simone, In a disconnected cluster - there is no operatorSource --> so your scripts tries to create it and failing (no connection to quay.io, etc)
(In reply to Asher Shoshan from comment #3) > Simone, > In a disconnected cluster - there is no operatorSource --> so your scripts > tries to create it and failing (no connection to quay.io, etc) did you already tried also with https://github.com/kubevirt/hyperconverged-cluster-operator/blob/master/deploy/deploy_imageregistry.sh pointing to the registry image you mirrored?
(In reply to Simone Tiraboschi from comment #6) > (In reply to Asher Shoshan from comment #3) > > Simone, > > In a disconnected cluster - there is no operatorSource --> so your scripts > > tries to create it and failing (no connection to quay.io, etc) > > did you already tried also with > https://github.com/kubevirt/hyperconverged-cluster-operator/blob/master/ > deploy/deploy_imageregistry.sh > > pointing to the registry image you mirrored? But again, this script creates catalogSource, which overrides the existing catalogSource (image pointing to local-registry).. This will pass, but pod will not come up (no internet connection..), and will mess up the disconnected catalogSource
Oren, can you please try reproducing this?
According to the step 4 in the official procedure: https://docs.openshift.com/container-platform/4.4/operators/olm-restricted-networks.html#olm-restricted-networks-operatorhub_olm-restricted-networks Create a CatalogSource object that references your catalog image. Modify the following to your specifications and save it as a catalogsource.yaml file: apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource metadata: name: my-operator-catalog namespace: openshift-marketplace spec: sourceType: grpc image: <registry_host_name>:<port>/olm/redhat-operators:v1 displayName: My Operator Catalog publisher: grpc Specify your custom Operator catalog image. Use the file to create the CatalogSource object: $ oc create -f catalogsource.yaml which is exactly what Oren is suggesting setting HCO_REGISTRY_IMAGE with the kustomize based deployment or using deploy_imageregistry.sh instead of deploy_marketplace.sh Am I missing the real issue?
MR has been merged to master branch.
deploy_kustomize.sh Failing : 13:29:23 ++ dirname cnv-qe-automation/cnv/deploy_kustomize.sh 13:29:23 + SCRIPT_DIR=/root/workspace/deploy-ocp-4.5-cnv-2.4-on-BM/cnv-qe-automation/cnv 13:29:23 + OC_TOOL=oc 13:29:23 + CNV_VERSION=2.4 13:29:23 + CNV_SOURCE=brew 13:29:23 + HCO_REGISTRY_IMAGE= 13:29:23 + CNV_SOURCES=(brew stage osbs production) 13:29:23 + CNV_VERSIONS=(2.4) 13:29:23 + SOURCES_MAP=(["brew"]="rh-verified-operators" ["stage"]="redhat-operators-stage" ["osbs"]="rh-osbs-operators" ["production"]="redhat-operators") 13:29:23 + declare -A SOURCES_MAP 13:29:23 + main 13:29:23 + fetch_upstream_bases 13:29:23 ++ mktemp -d 13:29:23 + TMPDIR=/tmp/tmp.wJec47evni 13:29:23 + trap 'rm -rf /tmp/tmp.wJec47evni' EXIT 13:29:23 + cd /tmp/tmp.wJec47evni 13:29:23 + curl -L https://api.github.com/repos/kubevirt/hyperconverged-cluster-operator/tarball/master 13:29:23 + tar --strip-components=1 -xvzf - 'kubevirt-hyperconverged-cluster-operator-*/deploy/kustomize' 13:29:23 % Total % Received % Xferd Average Speed Time Time Time Current 13:29:23 Dload Upload Total Spent Left Speed 13:29:23 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 13:29:23 kubevirt-hyperconverged-cluster-operator-0878c67/deploy/kustomize/ 13:29:23 kubevirt-hyperconverged-cluster-operator-0878c67/deploy/kustomize/README.md 13:29:23 kubevirt-hyperconverged-cluster-operator-0878c67/deploy/kustomize/base/ 13:29:23 kubevirt-hyperconverged-cluster-operator-0878c67/deploy/kustomize/base/hco_cr.yaml 13:29:23 kubevirt-hyperconverged-cluster-operator-0878c67/deploy/kustomize/base/kustomization.yaml 13:29:23 kubevirt-hyperconverged-cluster-operator-0878c67/deploy/kustomize/base/operator_group.yaml 13:29:23 kubevirt-hyperconverged-cluster-operator-0878c67/deploy/kustomize/base/subscription.yaml 13:29:23 kubevirt-hyperconverged-cluster-operator-0878c67/deploy/kustomize/deploy_kustomize.sh 13:29:23 kubevirt-hyperconverged-cluster-operator-0878c67/deploy/kustomize/image_registry/ 13:29:23 kubevirt-hyperconverged-cluster-operator-0878c67/deploy/kustomize/image_registry/catalog_source.yaml 13:29:23 kubevirt-hyperconverged-cluster-operator-0878c67/deploy/kustomize/image_registry/kustomization.yaml 13:29:23 kubevirt-hyperconverged-cluster-operator-0878c67/deploy/kustomize/kvm_emulation/ 13:29:23 kubevirt-hyperconverged-cluster-operator-0878c67/deploy/kustomize/kvm_emulation/kustomization.yaml 13:29:23 kubevirt-hyperconverged-cluster-operator-0878c67/deploy/kustomize/kvm_emulation/subscription.patch.yaml 13:29:23 kubevirt-hyperconverged-cluster-operator-0878c67/deploy/kustomize/marketplace/ 13:29:23 kubevirt-hyperconverged-cluster-operator-0878c67/deploy/kustomize/marketplace/kustomization.yaml 13:29:23 kubevirt-hyperconverged-cluster-operator-0878c67/deploy/kustomize/marketplace/operator_source.yaml 13:29:23 kubevirt-hyperconverged-cluster-operator-0878c67/deploy/kustomize/namespace.yaml 13:29:23 kubevirt-hyperconverged-cluster-operator-0878c67/deploy/kustomize/private_repo/ 13:29:23 kubevirt-hyperconverged-cluster-operator-0878c67/deploy/kustomize/private_repo/kustomization.yaml 13:29:23 kubevirt-hyperconverged-cluster-operator-0878c67/deploy/kustomize/private_repo/operator_source.patch.yaml 13:29:24 100 2337k 0 2337k 0 0 3915k 0 --:--:-- --:--:-- --:--:-- 3915k 100 7201k 0 7201k 0 0 4512k 0 --:--:-- 0:00:01 --:--:-- 4864k 100 9037k 0 9037k 0 0 4648k 0 --:--:-- 0:00:01 --:--:-- 4973k 13:29:24 + cd /root/workspace/deploy-ocp-4.5-cnv-2.4-on-BM/cnv-qe-automation/cnv 13:29:24 + cp -r /tmp/tmp.wJec47evni/deploy/kustomize /root/workspace/deploy-ocp-4.5-cnv-2.4-on-BM/cnv-qe-automation/cnv 13:29:24 + '[' -z ']' 13:29:24 + echo brew stage osbs production 13:29:24 + grep -q -w brew 13:29:24 + echo 2.4 13:29:24 + grep -q -w 2.4 13:29:24 + bash /root/workspace/deploy-ocp-4.5-cnv-2.4-on-BM/cnv-qe-automation/cnv/kustomize/cleanup_leftovers.sh 13:29:24 + export CLUSTER_DIR=/root/ocp/BM-savanna-cluster2 13:29:24 + CLUSTER_DIR=/root/ocp/BM-savanna-cluster2 13:29:24 + export KUBECONFIG=/root/ocp/BM-savanna-cluster2/auth/kubeconfig 13:29:24 + KUBECONFIG=/root/ocp/BM-savanna-cluster2/auth/kubeconfig 13:29:24 + for kind in vmi vm dv 13:29:24 + oc explain vmi 13:29:25 error: Couldn't find resource for "kubevirt.io/v1alpha3, Kind=VirtualMachineInstance" 13:29:25 + continue 13:29:25 + for kind in vmi vm dv 13:29:25 + oc explain vm 13:29:25 error: Couldn't find resource for "kubevirt.io/v1alpha3, Kind=VirtualMachine" 13:29:25 + continue 13:29:25 + for kind in vmi vm dv 13:29:25 + oc explain dv 13:29:26 error: Couldn't find resource for "cdi.kubevirt.io/v1alpha1, Kind=DataVolume" 13:29:26 + continue 13:29:26 + oc delete --ignore-not-found=true storageclass.storage.k8s.io/hostpath-provisioner 13:29:26 + oc delete hostpathprovisioner.hostpathprovisioner.kubevirt.io/hostpath-provisioner 13:29:26 Error from server (NotFound): hostpathprovisioners.hostpathprovisioner.kubevirt.io "hostpath-provisioner" not found 13:29:26 + true 13:29:26 ++ oc get catsrc rh-verified-operators -n openshift-marketplace -o=custom-columns=:metadata.name 13:29:26 + CAT_SRC=' 13:29:26 rh-verified-operators' 13:29:26 + '[' -n ' 13:29:26 rh-verified-operators' ']' 13:29:26 ++ oc get opsrc rh-verified-operators -n openshift-marketplace -o=custom-columns=:metadata.name 13:29:26 Error from server (NotFound): operatorsources.operators.coreos.com "rh-verified-operators" not found 13:29:26 + OP_SRC= 13:29:26 + true 13:29:26 + oc delete -k /root/workspace/deploy-ocp-4.5-cnv-2.4-on-BM/cnv-qe-automation/cnv/kustomize/brew 13:29:27 secret "quay-registry-kubevirt-hyperconverged" deleted 13:29:27 Error from server (NotFound): error when deleting "/root/workspace/deploy-ocp-4.5-cnv-2.4-on-BM/cnv-qe-automation/cnv/kustomize/brew": operatorsources.operators.coreos.com "kubevirt-hyperconverged" not found 13:29:27 + true 13:29:27 + sed -i 's/- namespace/#- namespace/' /root/workspace/deploy-ocp-4.5-cnv-2.4-on-BM/cnv-qe-automation/cnv/kustomize/deploy_hco_2.4/kustomization.yaml 13:29:27 + oc delete -k /root/workspace/deploy-ocp-4.5-cnv-2.4-on-BM/cnv-qe-automation/cnv/kustomize/deploy_hco_2.4 13:29:27 operatorgroup.operators.coreos.com "kubevirt-hyperconverged-group" deleted 13:29:27 Error from server (NotFound): error when deleting "/root/workspace/deploy-ocp-4.5-cnv-2.4-on-BM/cnv-qe-automation/cnv/kustomize/deploy_hco_2.4": subscriptions.operators.coreos.com "hco-operatorhub" not found 13:29:27 unable to recognize "/root/workspace/deploy-ocp-4.5-cnv-2.4-on-BM/cnv-qe-automation/cnv/kustomize/deploy_hco_2.4": no matches for kind "HyperConverged" in version "hco.kubevirt.io/v1beta1" 13:29:27 + true 13:29:27 + sed -i 's/#- namespace/- namespace/' /root/workspace/deploy-ocp-4.5-cnv-2.4-on-BM/cnv-qe-automation/cnv/kustomize/deploy_hco_2.4/kustomization.yaml 13:29:27 ++ oc get csv -n openshift-cnv --no-headers 13:29:27 ++ awk '{print $1}' 13:29:27 No resources found in openshift-cnv namespace. 13:29:27 + CSV_NAME= 13:29:27 + '[' -n '' ']' 13:29:27 + oc wait pod --for=delete -n openshift-cnv --all --timeout=60s 13:29:27 error: no matching resources found 13:29:27 + true 13:29:27 + echo '[INFO] Delivering HCO content to OLM' 13:29:27 [INFO] Delivering HCO content to OLM 13:29:27 + '[' -z ']' 13:29:27 + '[' -z ']' 13:29:27 + '[' -n ' 13:29:27 rh-verified-operators' ']' 13:29:27 + echo '[INFO]: Skipping creating an OperatorSource in a disconnected cluster.' 13:29:27 [INFO]: Skipping creating an OperatorSource in a disconnected cluster. 13:29:27 + sed -E -i 's|(source: ).*|\1 13:29:27 rh-verified-operators|' /root/workspace/deploy-ocp-4.5-cnv-2.4-on-BM/cnv-qe-automation/cnv/kustomize/deploy_hco_2.4/subscription.patch.yaml 13:29:27 sed: -e expression #1, char 17: unterminated `s' command 13:29:27 + rm -rf /tmp/tmp.wJec47evni
Hi Asher, Have you checked out from master branch of: https://gitlab.cee.redhat.com/contra/cnv-qe-automation/-/blob/master/cnv/deploy_kustomize.sh ? Because the master branch contains manifests for 2.5, and for CNV 2.4 you should checkout cnv-2.4-ocp-4.5 branch.
I checked out 2.4 (my branch based on this 2.4) - see also in log it's taking 2.4
Correct, the catalog source name was extracted erroneously with a carriage return, thus the error in `sed`. I submitted an MR which fixes it. https://gitlab.cee.redhat.com/contra/cnv-qe-automation/-/merge_requests/331
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:3194