Description of problem: On HCO deployment we failed to deploy SSP operator [root@dell-r640-008 ~]# oc describe pod kubevirt-ssp-operator-6f89bd7c46-pf956 -n kubevirt-hyperconverged Name: kubevirt-ssp-operator-6f89bd7c46-pf956 Namespace: kubevirt-hyperconverged Priority: 0 PriorityClassName: <none> Node: working-8mxhm-worker-0-7h55w/192.168.126.52 Start Time: Tue, 30 Apr 2019 17:51:37 +0300 Labels: name=kubevirt-ssp-operator pod-template-hash=6f89bd7c46 Annotations: alm-examples=[ { "apiVersion": "hco.kubevirt.io/v1alpha1", "kind": "HyperConverged", "metadata": { "name": "kubevirt-hyperconverged", "namespace": "kubevirt-hyperconverged" ... capabilities=Full Lifecycle categories=OpenShift Optional certified=false containerImage=brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/hyperconverged-cluster-operator:v2.0.0-9 createdAt=2019-04-17T16:01:59Z description=Creates and maintains a HyperConverged KubeVirt Deployment k8s.v1.cni.cncf.io/networks-status=[{ "name": "openshift-sdn", "interface": "eth0", "ips": [ "10.130.0.23" ], "default": true, "dns": {} }] olm.operatorGroup=hco-operatorgroup olm.operatorNamespace=kubevirt-hyperconverged olm.targetNamespaces= openshift.io/scc=anyuid repository=https://github.com/kubevirt/hyperconverged-cluster-operator support=false Status: Running IP: 10.130.0.23 Controlled By: ReplicaSet/kubevirt-ssp-operator-6f89bd7c46 Containers: kubevirt-ssp-operator: Container ID: cri-o://98f875cc21b2378a3036c362c660d3a331b33e5045feeaedd5b9d11359c0e1cc Image: brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/kubevirt-ssp-operator:v2.0.0-7 Image ID: brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/kubevirt-ssp-operator@sha256:fb56ca9ac65a69f7805455fe867989700872c42492e610cae616c55680a1f889 Port: 60000/TCP Host Port: 0/TCP State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Error Exit Code: 1 Started: Tue, 30 Apr 2019 22:12:59 +0300 Finished: Tue, 30 Apr 2019 22:13:00 +0300 Ready: False Restart Count: 55 Environment: WATCH_NAMESPACE: kubevirt-hyperconverged (v1:metadata.namespace) OPERATOR_NAME: kubevirt-ssp-operator Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kubevirt-ssp-operator-token-7qjf6 (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: kubevirt-ssp-operator-token-7qjf6: Type: Secret (a volume populated by a Secret) SecretName: kubevirt-ssp-operator-token-7qjf6 Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Pulled 41m (x48 over 4h) kubelet, working-8mxhm-worker-0-7h55w Successfully pulled image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/kubevirt-ssp-operator:v2.0.0-7" Normal Pulling 31m (x50 over 4h) kubelet, working-8mxhm-worker-0-7h55w Pulling image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/kubevirt-ssp-operator:v2.0.0-7" Warning BackOff 1m (x1169 over 4h) kubelet, working-8mxhm-worker-0-7h55w Back-off restarting failed container root@dell-r640-008 ~]# oc logs kubevirt-ssp-operator-6f89bd7c46-pf956 -n kubevirt-hyperconverged {"level":"info","ts":1556651267.9799497,"logger":"cmd","msg":"Go Version: go1.10.8"} {"level":"info","ts":1556651267.979987,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"} {"level":"info","ts":1556651267.9799957,"logger":"cmd","msg":"Version of operator-sdk: v0.7.0+git"} {"level":"info","ts":1556651267.9800034,"logger":"cmd","msg":"Go Version: go1.10.8"} {"level":"info","ts":1556651267.980006,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"} {"level":"info","ts":1556651267.9800084,"logger":"cmd","msg":"Version of operator-sdk: v0.7.0+git"} {"level":"info","ts":1556651267.9800231,"logger":"cmd","msg":"Watching namespace.","Namespace":"kubevirt-hyperconverged"} {"level":"info","ts":1556651268.1199892,"logger":"leader","msg":"Trying to become the leader."} {"level":"error","ts":1556651268.204345,"logger":"cmd","msg":"Failed to become leader.","Namespace":"kubevirt-hyperconverged","error":"required env POD_NAME not set, please configure downward API","stacktrace":"github.com/operator-framework/operator-sdk/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/operator-framework/operator-sdk/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/ansible.Run\n\t/go/src/github.com/operator-framework/operator-sdk/pkg/ansible/run.go:83\ngithub.com/operator-framework/operator-sdk/cmd/operator-sdk/run.newRunAnsibleCmd.func1\n\t/go/src/github.com/operator-framework/operator-sdk/cmd/operator-sdk/run/ansible.go:38\ngithub.com/operator-framework/operator-sdk/vendor/github.com/spf13/cobra.(*Command).execute\n\t/go/src/github.com/operator-framework/operator-sdk/vendor/github.com/spf13/cobra/command.go:762\ngithub.com/operator-framework/operator-sdk/vendor/github.com/spf13/cobra.(*Command).ExecuteC\n\t/go/src/github.com/operator-framework/operator-sdk/vendor/github.com/spf13/cobra/command.go:852\ngithub.com/operator-framework/operator-sdk/vendor/github.com/spf13/cobra.(*Command).Execute\n\t/go/src/github.com/operator-framework/operator-sdk/vendor/github.com/spf13/cobra/command.go:800\nmain.main\n\t/go/src/github.com/operator-framework/operator-sdk/cmd/operator-sdk/main.go:61\nruntime.main\n\t/opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/proc.go:198"} Error: required env POD_NAME not set, please configure downward API Usage: operator-sdk run ansible [flags] Flags: -h, --help help for ansible --inject-owner-ref The ansible operator will inject owner references unless this flag is false (default true) --max-workers int Maximum number of workers to use. Overridden by environment variable. (default 1) --reconcile-period duration Default reconcile period for controllers (default 1m0s) --watches-file string Path to the watches file to use (default "./watches.yaml") --zap-devel Enable zap development mode (changes defaults to console encoder, debug log level, and disables sampling) --zap-encoder encoder Zap log encoding ('json' or 'console') --zap-level level Zap log level (one of 'debug', 'info', 'error' or any integer value > 0) (default info) --zap-sample sample Enable zap log sampling. Sampling will be disabled for integer log levels > 1
Created attachment 1560492 [details] hco-operator info
Failed in build hco-bundle-registry:v2.0.0-13
Check again once kubevirt-ssp-operator-2.0.0-13 is available in HCO please. We just fixed a bunch of issues this morning.
the issue is fixed on hco:v2.0.0-13. # oc get pod kubevirt-ssp-operator-578cb75cbd-g77lk -n kubevirt-hyperconverged NAME READY STATUS RESTARTS AGE kubevirt-ssp-operator-578cb75cbd-g77lk 1/1 Running 0 29m # oc describe pod kubevirt-ssp-operator-578cb75cbd-g77lk -n kubevirt-hyperconverged Name: kubevirt-ssp-operator-578cb75cbd-g77lk Namespace: kubevirt-hyperconverged Priority: 0 PriorityClassName: <none> Node: working-lmsvw-worker-0-8jjnt/192.168.126.52 Start Time: Wed, 15 May 2019 23:54:15 -0400 Labels: name=kubevirt-ssp-operator pod-template-hash=578cb75cbd Annotations: alm-examples=[ { "apiVersion": "hco.kubevirt.io/v1alpha1", "kind": "HyperConverged", "metadata": { "name": "kubevirt-hyperconverged", "namespace": "kubevirt-hyperconverged" ... capabilities=Full Lifecycle categories=OpenShift Optional certified=false containerImage=brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/hyperconverged-cluster-operator:v2.0.0-20 createdAt=2019-04-17 16:01:59 description=Creates and maintains a HyperConverged KubeVirt Deployment k8s.v1.cni.cncf.io/networks-status=[{ "name": "openshift-sdn", "interface": "eth0", "ips": [ "10.130.0.40" ], "default": true, "dns": {} }] olm.operatorGroup=hco-operatorgroup olm.operatorNamespace=kubevirt-hyperconverged olm.targetNamespaces= openshift.io/scc=anyuid repository=https://github.com/kubevirt/hyperconverged-cluster-operator support=false Status: Running IP: 10.130.0.40 Controlled By: ReplicaSet/kubevirt-ssp-operator-578cb75cbd Containers: kubevirt-ssp-operator: Container ID: cri-o://666914acad673cdb897f1bee489a4fcaee83f552abd90eb27120774e898a34d8 Image: brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/kubevirt-ssp-operator:v2.0.0-10 Image ID: brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/kubevirt-ssp-operator@sha256:4b875c20180fc0fd7a8694e673140d3862113f15ee197aa4bdd6e434047086d1 Port: 60000/TCP Host Port: 0/TCP State: Running Started: Wed, 15 May 2019 23:57:07 -0400 Ready: True Restart Count: 0 Environment: POD_NAME: kubevirt-ssp-operator-578cb75cbd-g77lk (v1:metadata.name) WATCH_NAMESPACE: kubevirt-hyperconverged (v1:metadata.namespace) OPERATOR_NAME: kubevirt-ssp-operator Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kubevirt-ssp-operator-token-m4k2x (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: kubevirt-ssp-operator-token-m4k2x: Type: Secret (a volume populated by a Secret) SecretName: kubevirt-ssp-operator-token-m4k2x Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 29m default-scheduler Successfully assigned kubevirt-hyperconverged/kubevirt-ssp-operator-578cb75cbd-g77lk to working-lmsvw-worker-0-8jjnt Normal Pulling 29m kubelet, working-lmsvw-worker-0-8jjnt Pulling image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/kubevirt-ssp-operator:v2.0.0-10" Normal Pulled 26m kubelet, working-lmsvw-worker-0-8jjnt Successfully pulled image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/kubevirt-ssp-operator:v2.0.0-10" Normal Created 26m kubelet, working-lmsvw-worker-0-8jjnt Created container kubevirt-ssp-operator Normal Started 26m kubelet, working-lmsvw-worker-0-8jjnt Started container kubevirt-ssp-operator
Which installation you used: hco-bundle-registry:v2.0.0-13? If yes this build is still broken and missing a lot of pods, better to test with working installation.
Created attachment 1570053 [details] ssp on hco-bundle-registry:v2.0.0-15
@Israel, yes, that was on hco-bundle-registry:v2.0.0-13 For hco-bundle-registry:v2.0.0-15 ssp works too. See the attachment. Will it be ok or do you want to wait for another build?
Irina, Yes it not failing, what i want to verify that it contains all what it should, See https://bugzilla.redhat.com/show_bug.cgi?id=1709268#c10 looks like we need to verify the API also. About the common templates still missing but handles in diff BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1709268#c14
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:1850