Fedora Account System
Red Hat Associate
Red Hat Customer
Description of problem: After installation of hco-bundle-registry:v2.0.0-10 Common templates missing. [root@zeus07 ~]# oc get templates -n openshift | grep rhel [root@zeus07 ~]# [root@zeus07 ~]# oc get pod -n kubevirt-hyperconverged NAME READY STATUS RESTARTS AGE cdi-operator-5965b79d9-fwdsz 1/1 Running 2 22h cluster-network-addons-operator-6ffdb55c98-vbpxc 1/1 Running 0 22h hco-operator-65c4c48774-ntww5 1/1 Running 0 22h kubevirt-ssp-operator-bf8b698bd-rg8n8 1/1 Running 0 22h kubevirt-web-ui-operator-57f5c5ffbf-h4j76 1/1 Running 0 22h node-maintenance-operator-c57d546f8-jmcwq 1/1 Running 0 22h virt-api-8bdf97f8c-4xmcj 1/1 Running 0 22h virt-api-8bdf97f8c-rn5qm 1/1 Running 0 22h virt-controller-8b6454cc9-pk5dt 1/1 Running 2 22h virt-controller-8b6454cc9-tjkml 1/1 Running 0 22h virt-handler-9qmxx 1/1 Running 0 22h virt-handler-fcx9j 1/1 Running 0 22h virt-operator-594b845cf6-f8jdw 1/1 Running 2 22h virt-operator-594b845cf6-zsvcd 1/1 Running 0 22h
Israel, can you attach the version and logs from the ssp operator?
Created attachment 1567943 [details] ssp-operator info
version - image: "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/kubevirt-ssp-operator:v2.0.0-9" [root@dell-r640-008 ~]# oc describe pod/kubevirt-ssp-operator-bf8b698bd-pnkqg -n kubevirt-hyperconverged Name: kubevirt-ssp-operator-bf8b698bd-pnkqg Namespace: kubevirt-hyperconverged Priority: 0 PriorityClassName: <none> Node: working-hsd77-worker-0-vkwkt/192.168.126.53 Start Time: Mon, 13 May 2019 14:04:34 +0300 Labels: name=kubevirt-ssp-operator pod-template-hash=bf8b698bd Annotations: alm-examples=[ { "apiVersion": "hco.kubevirt.io/v1alpha1", "kind": "HyperConverged", "metadata": { "name": "kubevirt-hyperconverged", "namespace": "kubevirt-hyperconverged" ... capabilities=Full Lifecycle categories=OpenShift Optional certified=false containerImage=brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/hyperconverged-cluster-operator:v2.0.0-17 createdAt=2019-04-17 16:01:59 description=Creates and maintains a HyperConverged KubeVirt Deployment k8s.v1.cni.cncf.io/networks-status=[{ "name": "openshift-sdn", "interface": "eth0", "ips": [ "10.131.0.35" ], "default": true, "dns": {} }] olm.operatorGroup=hco-operatorgroup olm.operatorNamespace=kubevirt-hyperconverged olm.targetNamespaces= openshift.io/scc=anyuid repository=https://github.com/kubevirt/hyperconverged-cluster-operator support=false Status: Running IP: 10.131.0.35 Controlled By: ReplicaSet/kubevirt-ssp-operator-bf8b698bd Containers: kubevirt-ssp-operator: Container ID: cri-o://a81a0655888263c071814d446ed2cd1744aa7c13889d9e38dbc68e75adf47e78 Image: brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/kubevirt-ssp-operator:v2.0.0-9 Image ID: brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/kubevirt-ssp-operator@sha256:c940a0ca3063a10bc755be2e11f810b0646fd2bc9d75a077de417cd2eccb3554 Port: 60000/TCP Host Port: 0/TCP State: Running Started: Mon, 13 May 2019 14:06:54 +0300 Ready: True Restart Count: 0 Environment: POD_NAME: kubevirt-ssp-operator-bf8b698bd-pnkqg (v1:metadata.name) WATCH_NAMESPACE: kubevirt-hyperconverged (v1:metadata.namespace) OPERATOR_NAME: kubevirt-ssp-operator Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kubevirt-ssp-operator-token-dvd9h (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: kubevirt-ssp-operator-token-dvd9h: Type: Secret (a volume populated by a Secret) SecretName: kubevirt-ssp-operator-token-dvd9h Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 1h default-scheduler Successfully assigned kubevirt-hyperconverged/kubevirt-ssp-operator-bf8b698bd-pnkqg to working-hsd77-worker-0-vkwkt Normal Pulling 1h kubelet, working-hsd77-worker-0-vkwkt Pulling image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/kubevirt-ssp-operator:v2.0.0-9" Normal Pulled 1h kubelet, working-hsd77-worker-0-vkwkt Successfully pulled image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/kubevirt-ssp-operator:v2.0.0-9" Normal Created 1h kubelet, working-hsd77-worker-0-vkwkt Created container kubevirt-ssp-operator Normal Started 1h kubelet, working-hsd77-worker-0-vkwkt Started container kubevirt-ssp-operator
Looks like this is caused by: /opt/ansible/roles/KubevirtNodeLabeller/tasks/main.yml:2\u001b[0m\n\u001b[0;31mfatal: [localhost]: FAILED! => {\"msg\": \"The task includes an option with an undefined variable. The error was: 'version' is undefined Karel, can you take a look please?
For starters, we will need https://github.com/MarSik/kubevirt-ssp-operator/pull/36 - and the HCO counterpart. Investigation still ongoing.
We need to learn if ssp-operator broke under HCO (on its own seems to work fine), and if it deploys the templates in the right namespace.
This is the likely case: https://github.com/MarSik/kubevirt-ssp-operator/pull/36/commits/f6dd617644314f2d42a1d913b34d44bee5dce7f9 - the hco op uses (and it is the only consumer) the ssp-op go api to create the CR - the CR is serialized in the wrong way To fix the above issue, we need 1. to commit the PR 2. to add a new tag 3. to update HCO to vendor the new tag, then rebuild It is yet to be understood why the ssp-op don't use the default versions for the common templates
(In reply to Francesco Romani from comment #9) > This is the likely case: > https://github.com/MarSik/kubevirt-ssp-operator/pull/36/commits/ > f6dd617644314f2d42a1d913b34d44bee5dce7f9 > - the hco op uses (and it is the only consumer) the ssp-op go api to create > the CR > - the CR is serialized in the wrong way > > To fix the above issue, we need > 1. to commit the PR > 2. to add a new tag > 3. to update HCO to vendor the new tag, then rebuild > > It is yet to be understood why the ssp-op don't use the default versions for > the common templates SSP operator fixed, image here: https://quay.io/repository/fromani/kubevirt-ssp-operator-container HCO PR: https://github.com/kubevirt/hyperconverged-cluster-operator/pull/91/
form the operator log: # oc logs kubevirt-ssp-operator-99f67b8cd-m5tb7 -n kubevirt-hyperconverged | grep err {"level":"error","ts":1558355771.757527,"logger":"reconciler","msg":"Unable to update the status to mark cr as running","job":"4936049823809335360","name":"template-validator-hyperconverged-cluster","namespace":"kubevirt-hyperconverged","error":"Operation cannot be fulfilled on kubevirttemplatevalidators.kubevirt.io \"template-validator-hyperconverged-cluster\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/operator-framework/operator-sdk/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/operator-framework/operator-sdk/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/ansible/controller.(*AnsibleOperatorReconciler).Reconcile\n\t/go/src/github.com/operator-framework/operator-sdk/pkg/ansible/controller/reconcile.go:127\ngithub.com/operator-framework/operator-sdk/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/operator-framework/operator-sdk/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215\ngithub.com/operator-framework/operator-sdk/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/operator-framework/operator-sdk/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/operator-framework/operator-sdk/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/operator-framework/operator-sdk/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/operator-framework/operator-sdk/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/operator-framework/operator-sdk/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/operator-framework/operator-sdk/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/operator-framework/operator-sdk/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"} {"level":"error","ts":1558355771.7577739,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"kubevirttemplatevalidator-controller","request":"kubevirt-hyperconverged/template-validator-hyperconverged-cluster","error":"Operation cannot be fulfilled on kubevirttemplatevalidators.kubevirt.io \"template-validator-hyperconverged-cluster\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/operator-framework/operator-sdk/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/operator-framework/operator-sdk/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/operator-framework/operator-sdk/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/operator-framework/operator-sdk/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/operator-framework/operator-sdk/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/operator-framework/operator-sdk/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/operator-framework/operator-sdk/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/operator-framework/operator-sdk/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/operator-framework/operator-sdk/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/operator-framework/operator-sdk/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/operator-framework/operator-sdk/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/operator-framework/operator-sdk/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
Created attachment 1571277 [details] ssp-operator info - 20/5/19
A simple fix in the deployment manifest should be sufficient
(In reply to Francesco Romani from comment #18) > A simple fix in the deployment manifest should be sufficient Failed QE, waiting for new build ^^
HCO fix: https://github.com/kubevirt/hyperconverged-cluster-operator/pull/98 SSP fix: https://github.com/MarSik/kubevirt-ssp-operator/pull/43 no image rebuild needed.
All PRs merged
Verify with: hco-bundle-registry:v2.0.0-17 All templates exists, RHEL and windows view: [root@puma56 ~]# oc get template -n openshift | grep rhel rhel6-desktop-large This template can be used to create a VM suitable for Red Hat Enterprise Linu... 2 (1 blank) 1 rhel6-desktop-medium This template can be used to create a VM suitable for Red Hat Enterprise Linu... 2 (1 blank) 1 rhel6-desktop-small This template can be used to create a VM suitable for Red Hat Enterprise Linu... 2 (1 blank) 1 rhel6-desktop-tiny This template can be used to create a VM suitable for Red Hat Enterprise Linu... 2 (1 blank) 1 rhel6-server-large This template can be used to create a VM suitable for Red Hat Enterprise Linu... 2 (1 blank) 1 rhel6-server-medium This template can be used to create a VM suitable for Red Hat Enterprise Linu... 2 (1 blank) 1 rhel6-server-small This template can be used to create a VM suitable for Red Hat Enterprise Linu... 2 (1 blank) 1 rhel6-server-tiny This template can be used to create a VM suitable for Red Hat Enterprise Linu... 2 (1 blank) 1 rhel7-desktop-large This template can be used to create a VM suitable for Red Hat Enterprise Linu... 2 (1 blank) 1 rhel7-desktop-medium This template can be used to create a VM suitable for Red Hat Enterprise Linu... 2 (1 blank) 1 rhel7-desktop-small This template can be used to create a VM suitable for Red Hat Enterprise Linu... 2 (1 blank) 1 rhel7-desktop-tiny This template can be used to create a VM suitable for Red Hat Enterprise Linu... 2 (1 blank) 1 rhel7-highperformance-large This template can be used to create a VM suitable for Red Hat Enterprise Linu... 2 (1 blank) 1 rhel7-highperformance-medium This template can be used to create a VM suitable for Red Hat Enterprise Linu... 2 (1 blank) 1 rhel7-highperformance-small This template can be used to create a VM suitable for Red Hat Enterprise Linu... 2 (1 blank) 1 rhel7-highperformance-tiny This template can be used to create a VM suitable for Red Hat Enterprise Linu... 2 (1 blank) 1 rhel7-server-large This template can be used to create a VM suitable for Red Hat Enterprise Linu... 2 (1 blank) 1 rhel7-server-medium This template can be used to create a VM suitable for Red Hat Enterprise Linu... 2 (1 blank) 1 rhel7-server-small This template can be used to create a VM suitable for Red Hat Enterprise Linu... 2 (1 blank) 1 rhel7-server-tiny This template can be used to create a VM suitable for Red Hat Enterprise Linu... 2 (1 blank) 1 rhel8-desktop-large This template can be used to create a VM suitable for Red Hat Enterprise Linu... 2 (1 blank) 1 rhel8-desktop-medium This template can be used to create a VM suitable for Red Hat Enterprise Linu... 2 (1 blank) 1 rhel8-desktop-small This template can be used to create a VM suitable for Red Hat Enterprise Linu... 2 (1 blank) 1 rhel8-desktop-tiny This template can be used to create a VM suitable for Red Hat Enterprise Linu... 2 (1 blank) 1 rhel8-highperformance-large This template can be used to create a VM suitable for Red Hat Enterprise Linu... 2 (1 blank) 1 rhel8-highperformance-medium This template can be used to create a VM suitable for Red Hat Enterprise Linu... 2 (1 blank) 1 rhel8-highperformance-small This template can be used to create a VM suitable for Red Hat Enterprise Linu... 2 (1 blank) 1 rhel8-highperformance-tiny This template can be used to create a VM suitable for Red Hat Enterprise Linu... 2 (1 blank) 1 rhel8-server-large This template can be used to create a VM suitable for Red Hat Enterprise Linu... 2 (1 blank) 1 rhel8-server-medium This template can be used to create a VM suitable for Red Hat Enterprise Linu... 2 (1 blank) 1 rhel8-server-small This template can be used to create a VM suitable for Red Hat Enterprise Linu... 2 (1 blank) 1 rhel8-server-tiny This template can be used to create a VM suitable for Red Hat Enterprise Linu... 2 (1 blank) 1 [root@puma56 ~]# oc get template -n openshift | grep win win2k12r2-desktop-large This template can be used to create a VM suitable for Microsoft Windows Serve... 2 (1 blank) 1 win2k12r2-desktop-medium This template can be used to create a VM suitable for Microsoft Windows Serve... 2 (1 blank) 1 win2k12r2-server-large This template can be used to create a VM suitable for Microsoft Windows Serve... 2 (1 blank) 1 win2k12r2-server-medium This template can be used to create a VM suitable for Microsoft Windows Serve... 2 (1 blank) 1