Bug 1704914 - SSP operator failed to deploy
Summary: SSP operator failed to deploy
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Container Native Virtualization (CNV)
Classification: Red Hat
Component: Installation
Version: 2.0
Hardware: Unspecified
OS: Unspecified
high
urgent
Target Milestone: ---
: 2.0
Assignee: Martin Sivák
QA Contact: Irina Gulina
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-04-30 19:14 UTC by Israel Pinto
Modified: 2019-07-24 20:16 UTC (History)
7 users (show)

Fixed In Version: hyperconverged-cluster-operator-container-v2.0.0-12
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-07-24 20:15:51 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
hco-operator info (114.01 KB, text/plain)
2019-04-30 19:20 UTC, Israel Pinto
no flags Details
ssp on hco-bundle-registry:v2.0.0-15 (5.16 KB, text/plain)
2019-05-17 11:50 UTC, Irina Gulina
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2019:1850 0 None None None 2019-07-24 20:15:59 UTC

Description Israel Pinto 2019-04-30 19:14:24 UTC
Description of problem:
On HCO deployment we failed to deploy SSP operator

[root@dell-r640-008 ~]# oc describe pod kubevirt-ssp-operator-6f89bd7c46-pf956 -n kubevirt-hyperconverged
Name:               kubevirt-ssp-operator-6f89bd7c46-pf956
Namespace:          kubevirt-hyperconverged
Priority:           0
PriorityClassName:  <none>
Node:               working-8mxhm-worker-0-7h55w/192.168.126.52
Start Time:         Tue, 30 Apr 2019 17:51:37 +0300
Labels:             name=kubevirt-ssp-operator
                    pod-template-hash=6f89bd7c46
Annotations:        alm-examples=[
  {
    "apiVersion": "hco.kubevirt.io/v1alpha1",
    "kind": "HyperConverged",
    "metadata": {
      "name": "kubevirt-hyperconverged",
      "namespace": "kubevirt-hyperconverged"
 ...
  capabilities=Full Lifecycle
  categories=OpenShift Optional
  certified=false
  containerImage=brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/hyperconverged-cluster-operator:v2.0.0-9
  createdAt=2019-04-17T16:01:59Z
  description=Creates and maintains a HyperConverged KubeVirt Deployment
  k8s.v1.cni.cncf.io/networks-status=[{
    "name": "openshift-sdn",
    "interface": "eth0",
    "ips": [
        "10.130.0.23"
    ],
    "default": true,
    "dns": {}
}]
                olm.operatorGroup=hco-operatorgroup
                olm.operatorNamespace=kubevirt-hyperconverged
                olm.targetNamespaces=
                openshift.io/scc=anyuid
                repository=https://github.com/kubevirt/hyperconverged-cluster-operator
                support=false
Status:         Running
IP:             10.130.0.23
Controlled By:  ReplicaSet/kubevirt-ssp-operator-6f89bd7c46
Containers:
  kubevirt-ssp-operator:
    Container ID:   cri-o://98f875cc21b2378a3036c362c660d3a331b33e5045feeaedd5b9d11359c0e1cc
    Image:          brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/kubevirt-ssp-operator:v2.0.0-7
    Image ID:       brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/kubevirt-ssp-operator@sha256:fb56ca9ac65a69f7805455fe867989700872c42492e610cae616c55680a1f889
    Port:           60000/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Tue, 30 Apr 2019 22:12:59 +0300
      Finished:     Tue, 30 Apr 2019 22:13:00 +0300
    Ready:          False
    Restart Count:  55
    Environment:
      WATCH_NAMESPACE:  kubevirt-hyperconverged (v1:metadata.namespace)
      OPERATOR_NAME:    kubevirt-ssp-operator
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kubevirt-ssp-operator-token-7qjf6 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  kubevirt-ssp-operator-token-7qjf6:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  kubevirt-ssp-operator-token-7qjf6
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason   Age                 From                                   Message
  ----     ------   ----                ----                                   -------
  Normal   Pulled   41m (x48 over 4h)   kubelet, working-8mxhm-worker-0-7h55w  Successfully pulled image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/kubevirt-ssp-operator:v2.0.0-7"
  Normal   Pulling  31m (x50 over 4h)   kubelet, working-8mxhm-worker-0-7h55w  Pulling image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/kubevirt-ssp-operator:v2.0.0-7"
  Warning  BackOff  1m (x1169 over 4h)  kubelet, working-8mxhm-worker-0-7h55w  Back-off restarting failed container
 

root@dell-r640-008 ~]# oc logs kubevirt-ssp-operator-6f89bd7c46-pf956 -n kubevirt-hyperconverged
{"level":"info","ts":1556651267.9799497,"logger":"cmd","msg":"Go Version: go1.10.8"}
{"level":"info","ts":1556651267.979987,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"}
{"level":"info","ts":1556651267.9799957,"logger":"cmd","msg":"Version of operator-sdk: v0.7.0+git"}
{"level":"info","ts":1556651267.9800034,"logger":"cmd","msg":"Go Version: go1.10.8"}
{"level":"info","ts":1556651267.980006,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"}
{"level":"info","ts":1556651267.9800084,"logger":"cmd","msg":"Version of operator-sdk: v0.7.0+git"}
{"level":"info","ts":1556651267.9800231,"logger":"cmd","msg":"Watching namespace.","Namespace":"kubevirt-hyperconverged"}
{"level":"info","ts":1556651268.1199892,"logger":"leader","msg":"Trying to become the leader."}
{"level":"error","ts":1556651268.204345,"logger":"cmd","msg":"Failed to become leader.","Namespace":"kubevirt-hyperconverged","error":"required env POD_NAME not set, please configure downward API","stacktrace":"github.com/operator-framework/operator-sdk/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/operator-framework/operator-sdk/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/ansible.Run\n\t/go/src/github.com/operator-framework/operator-sdk/pkg/ansible/run.go:83\ngithub.com/operator-framework/operator-sdk/cmd/operator-sdk/run.newRunAnsibleCmd.func1\n\t/go/src/github.com/operator-framework/operator-sdk/cmd/operator-sdk/run/ansible.go:38\ngithub.com/operator-framework/operator-sdk/vendor/github.com/spf13/cobra.(*Command).execute\n\t/go/src/github.com/operator-framework/operator-sdk/vendor/github.com/spf13/cobra/command.go:762\ngithub.com/operator-framework/operator-sdk/vendor/github.com/spf13/cobra.(*Command).ExecuteC\n\t/go/src/github.com/operator-framework/operator-sdk/vendor/github.com/spf13/cobra/command.go:852\ngithub.com/operator-framework/operator-sdk/vendor/github.com/spf13/cobra.(*Command).Execute\n\t/go/src/github.com/operator-framework/operator-sdk/vendor/github.com/spf13/cobra/command.go:800\nmain.main\n\t/go/src/github.com/operator-framework/operator-sdk/cmd/operator-sdk/main.go:61\nruntime.main\n\t/opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/proc.go:198"}
Error: required env POD_NAME not set, please configure downward API
Usage:
  operator-sdk run ansible [flags]

Flags:
  -h, --help                        help for ansible
      --inject-owner-ref            The ansible operator will inject owner references unless this flag is false (default true)
      --max-workers int             Maximum number of workers to use. Overridden by environment variable. (default 1)
      --reconcile-period duration   Default reconcile period for controllers (default 1m0s)
      --watches-file string         Path to the watches file to use (default "./watches.yaml")
      --zap-devel                   Enable zap development mode (changes defaults to console encoder, debug log level, and disables sampling)
      --zap-encoder encoder         Zap log encoding ('json' or 'console')
      --zap-level level             Zap log level (one of 'debug', 'info', 'error' or any integer value > 0) (default info)
      --zap-sample sample           Enable zap log sampling. Sampling will be disabled for integer log levels > 1

Comment 1 Israel Pinto 2019-04-30 19:20:05 UTC
Created attachment 1560492 [details]
hco-operator info

Comment 3 Israel Pinto 2019-05-15 13:02:20 UTC
Failed in build hco-bundle-registry:v2.0.0-13

Comment 4 Martin Sivák 2019-05-15 13:04:39 UTC
Check again once kubevirt-ssp-operator-2.0.0-13 is available in HCO please. We just fixed a bunch of issues this morning.

Comment 7 Guohua Ouyang 2019-05-16 04:27:56 UTC
the issue is fixed on hco:v2.0.0-13.

# oc get pod kubevirt-ssp-operator-578cb75cbd-g77lk -n kubevirt-hyperconverged 
NAME                                     READY     STATUS    RESTARTS   AGE
kubevirt-ssp-operator-578cb75cbd-g77lk   1/1       Running   0          29m

# oc describe pod kubevirt-ssp-operator-578cb75cbd-g77lk -n kubevirt-hyperconverged 
Name:               kubevirt-ssp-operator-578cb75cbd-g77lk
Namespace:          kubevirt-hyperconverged
Priority:           0
PriorityClassName:  <none>
Node:               working-lmsvw-worker-0-8jjnt/192.168.126.52
Start Time:         Wed, 15 May 2019 23:54:15 -0400
Labels:             name=kubevirt-ssp-operator
                    pod-template-hash=578cb75cbd
Annotations:        alm-examples=[
  {
    "apiVersion": "hco.kubevirt.io/v1alpha1",
    "kind": "HyperConverged",
    "metadata": {
      "name": "kubevirt-hyperconverged",
      "namespace": "kubevirt-hyperconverged"
 ...
  capabilities=Full Lifecycle
  categories=OpenShift Optional
  certified=false
  containerImage=brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/hyperconverged-cluster-operator:v2.0.0-20
  createdAt=2019-04-17 16:01:59
  description=Creates and maintains a HyperConverged KubeVirt Deployment
  k8s.v1.cni.cncf.io/networks-status=[{
    "name": "openshift-sdn",
    "interface": "eth0",
    "ips": [
        "10.130.0.40"
    ],
    "default": true,
    "dns": {}
}]
                olm.operatorGroup=hco-operatorgroup
                olm.operatorNamespace=kubevirt-hyperconverged
                olm.targetNamespaces=
                openshift.io/scc=anyuid
                repository=https://github.com/kubevirt/hyperconverged-cluster-operator
                support=false
Status:         Running
IP:             10.130.0.40
Controlled By:  ReplicaSet/kubevirt-ssp-operator-578cb75cbd
Containers:
  kubevirt-ssp-operator:
    Container ID:   cri-o://666914acad673cdb897f1bee489a4fcaee83f552abd90eb27120774e898a34d8
    Image:          brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/kubevirt-ssp-operator:v2.0.0-10
    Image ID:       brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/kubevirt-ssp-operator@sha256:4b875c20180fc0fd7a8694e673140d3862113f15ee197aa4bdd6e434047086d1
    Port:           60000/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Wed, 15 May 2019 23:57:07 -0400
    Ready:          True
    Restart Count:  0
    Environment:
      POD_NAME:         kubevirt-ssp-operator-578cb75cbd-g77lk (v1:metadata.name)
      WATCH_NAMESPACE:  kubevirt-hyperconverged (v1:metadata.namespace)
      OPERATOR_NAME:    kubevirt-ssp-operator
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kubevirt-ssp-operator-token-m4k2x (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  kubevirt-ssp-operator-token-m4k2x:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  kubevirt-ssp-operator-token-m4k2x
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age   From                                   Message
  ----    ------     ----  ----                                   -------
  Normal  Scheduled  29m   default-scheduler                      Successfully assigned kubevirt-hyperconverged/kubevirt-ssp-operator-578cb75cbd-g77lk to working-lmsvw-worker-0-8jjnt
  Normal  Pulling    29m   kubelet, working-lmsvw-worker-0-8jjnt  Pulling image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/kubevirt-ssp-operator:v2.0.0-10"
  Normal  Pulled     26m   kubelet, working-lmsvw-worker-0-8jjnt  Successfully pulled image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/kubevirt-ssp-operator:v2.0.0-10"
  Normal  Created    26m   kubelet, working-lmsvw-worker-0-8jjnt  Created container kubevirt-ssp-operator
  Normal  Started    26m   kubelet, working-lmsvw-worker-0-8jjnt  Started container kubevirt-ssp-operator

Comment 9 Israel Pinto 2019-05-16 12:14:58 UTC
Which installation you used: hco-bundle-registry:v2.0.0-13?
If yes this build is still broken and missing a lot of pods, better to test with working  installation.

Comment 10 Irina Gulina 2019-05-17 11:50:21 UTC
Created attachment 1570053 [details]
ssp on hco-bundle-registry:v2.0.0-15

Comment 11 Irina Gulina 2019-05-17 11:51:41 UTC
@Israel, yes, that was on hco-bundle-registry:v2.0.0-13
For hco-bundle-registry:v2.0.0-15 ssp works too. See the attachment. Will it be ok or do you want to wait for another build?

Comment 12 Israel Pinto 2019-05-19 08:41:34 UTC
Irina,
Yes it not failing, what i want to verify that it contains all what it should,
See https://bugzilla.redhat.com/show_bug.cgi?id=1709268#c10 
looks like we need to verify the API also.

About the common templates still missing but handles in diff BZ: 
https://bugzilla.redhat.com/show_bug.cgi?id=1709268#c14

Comment 14 errata-xmlrpc 2019-07-24 20:15:51 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:1850


Note You need to log in before you can comment on or make changes to this bug.