Attachment 1606916 Details for Bug 1744534 – virt-operator pod describe output

virt-operator pod describe output

virt-operator_restarts_often.txt (text/plain), 20.41 KB, created by Kedar Bidarkar on 2019-08-22 11:14:28 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Kedar Bidarkar

Created: 2019-08-22 11:14:28 UTC

Size: 20.41 KB

patch

obsolete

>[kbidarka@kbidarka-host osdc]$ oc get pods -n openshift-cnv 
>NAME                                            READY     STATUS             RESTARTS   AGE
>cdi-operator-558b9746ff-q58gn                   0/1       CrashLoopBackOff   67         17h
>cluster-network-addons-operator-8697648-s6dfk   1/1       Running            0          17h
>hco-operator-54d874bf95-4msnv                   0/1       Running            0          17h
>kubevirt-ssp-operator-6dfdd49c54-lxq2f          0/1       CrashLoopBackOff   25         17h
>machine-remediation-operator-d84474b4d-m2ww4    1/1       Running            0          19h
>node-maintenance-operator-79b86cc7bd-hljpq      1/1       Running            0          19h
>virt-api-547fc86c6f-jz48c                       1/1       Running            0          9m43s
>virt-api-547fc86c6f-mc654                       1/1       Running            0          9m44s
>virt-controller-dfb67f66-7pdpq                  0/1       Error              0          9m23s
>virt-controller-dfb67f66-rltnz                  1/1       Running            3          9m24s
>virt-handler-gcvx6                              1/1       Running            0          9m23s
>virt-handler-qgvcv                              1/1       Running            0          9m23s
>virt-handler-slbpw                              1/1       Running            0          9m23s
>virt-operator-6c65f86c4f-gsdgd                  0/1       CrashLoopBackOff   45         17h
>virt-operator-6c65f86c4f-jhcsj                  0/1       CrashLoopBackOff   56         17h
>[kbidarka@kbidarka-host osdc]$ oc describe pod virt-operator-6c65f86c4f-gsdgd -n openshift-cnv 
>Name:               virt-operator-6c65f86c4f-gsdgd
>Namespace:          openshift-cnv
>Priority:           2000000000
>PriorityClassName:  system-cluster-critical
>Node:               host-172-16-0-23/172.16.0.23
>Start Time:         Wed, 21 Aug 2019 22:48:18 +0530
>Labels:             kubevirt.io=virt-operator
>                    pod-template-hash=6c65f86c4f
>                    prometheus.kubevirt.io=
>Annotations:        alm-examples=[
>  {
>    "apiVersion": "hco.kubevirt.io/v1alpha1",
>    "kind": "HyperConverged",
>    "metadata": {
>      "name": "kubevirt-hyperconverged",
>      "namespace": "kubevirt-hyperconverged"
> ...
>                capabilities=Full Lifecycle
>                categories=OpenShift Optional
>                certified=false
>                containerImage=brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/container-native-virtualization/hyperconverged-cluster-operator:v2.1.0-6
>                createdAt=2019-08-21 15:44:09.993271
>                description=Creates and maintains a HyperConverged KubeVirt Deployment
>                olm.operatorGroup=hco-operatorgroup
>                olm.operatorNamespace=openshift-cnv
>                olm.targetNamespaces=
>                openshift.io/scc=restricted
>                repository=https://github.com/kubevirt/hyperconverged-cluster-operator
>                scheduler.alpha.kubernetes.io/critical-pod=
>                scheduler.alpha.kubernetes.io/tolerations=[{"key":"CriticalAddonsOnly","operator":"Exists"}]
>                support=false
>Status:         Running
>IP:             10.128.0.57
>Controlled By:  ReplicaSet/virt-operator-6c65f86c4f
>Containers:
>  virt-operator:
>    Container ID:  cri-o://5f075db453d2725e4fa08bf152a30432e97f79ef3fe2f0538dcc55c70f12119c
>    Image:         brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/virt-operator:v2.1.0-7
>    Image ID:      brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/virt-operator@sha256:7fc9386a6a307b60b190c866f84a375a05ee3b26b4b0eeb5867c1673a28c708b
>    Port:          8443/TCP
>    Host Port:     0/TCP
>    Command:
>      virt-operator
>      --port
>      8443
>      -v
>      2
>    State:          Waiting
>      Reason:       CrashLoopBackOff
>    Last State:     Terminated
>      Reason:       Error
>      Exit Code:    1
>      Started:      Thu, 22 Aug 2019 16:09:13 +0530
>      Finished:     Thu, 22 Aug 2019 16:12:18 +0530
>    Ready:          False
>    Restart Count:  45
>    Readiness:      http-get https://:8443/metrics delay=5s timeout=10s period=10s #success=1 #failure=3
>    Environment:
>      OPERATOR_IMAGE:   brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/virt-operator:v2.1.0-7
>      WATCH_NAMESPACE:   (v1:metadata.annotations['olm.targetNamespaces'])
>    Mounts:
>      /var/run/secrets/kubernetes.io/serviceaccount from kubevirt-operator-token-kp44c (ro)
>Conditions:
>  Type              Status
>  Initialized       True 
>  Ready             False 
>  ContainersReady   False 
>  PodScheduled      True 
>Volumes:
>  kubevirt-operator-token-kp44c:
>    Type:        Secret (a volume populated by a Secret)
>    SecretName:  kubevirt-operator-token-kp44c
>    Optional:    false
>QoS Class:       BestEffort
>Node-Selectors:  <none>
>Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
>                 node.kubernetes.io/unreachable:NoExecute for 300s
>Events:
>  Type     Reason       Age                 From                       Message
>  ----     ------       ----                ----                       -------
>  Warning  FailedMount  2h                  kubelet, host-172-16-0-23  MountVolume.SetUp failed for volume "kubevirt-operator-token-kp44c" : Get https://api-int.working.oc4:6443/api/v1/namespaces/openshift-cnv/secrets/kubevirt-operator-token-kp44c: read tcp 172.16.0.23:38342->10.0.96.108:6443: use of closed network connection
>  Warning  FailedMount  2h (x8 over 9h)     kubelet, host-172-16-0-23  MountVolume.SetUp failed for volume "kubevirt-operator-token-kp44c" : secrets "kubevirt-operator-token-kp44c" is forbidden: User "system:node:host-172-16-0-23" cannot get resource "secrets" in API group "" in the namespace "openshift-cnv": no relationship found between node "host-172-16-0-23" and this object
>  Warning  FailedMount  2h                  kubelet, host-172-16-0-23  MountVolume.SetUp failed for volume "kubevirt-operator-token-kp44c" : Get https://api-int.working.oc4:6443/api/v1/namespaces/openshift-cnv/secrets/kubevirt-operator-token-kp44c: read tcp 172.16.0.23:51476->10.0.96.108:6443: use of closed network connection
>  Warning  FailedMount  24m                 kubelet, host-172-16-0-23  MountVolume.SetUp failed for volume "kubevirt-operator-token-kp44c" : Get https://api-int.working.oc4:6443/api/v1/namespaces/openshift-cnv/secrets/kubevirt-operator-token-kp44c: read tcp 172.16.0.23:36204->10.0.96.108:6443: use of closed network connection
>  Warning  FailedMount  24m                 kubelet, host-172-16-0-23  MountVolume.SetUp failed for volume "kubevirt-operator-token-kp44c" : Get https://api-int.working.oc4:6443/api/v1/namespaces/openshift-cnv/secrets/kubevirt-operator-token-kp44c: read tcp 172.16.0.23:36302->10.0.96.108:6443: use of closed network connection
>  Warning  FailedMount  24m                 kubelet, host-172-16-0-23  MountVolume.SetUp failed for volume "kubevirt-operator-token-kp44c" : Get https://api-int.working.oc4:6443/api/v1/namespaces/openshift-cnv/secrets/kubevirt-operator-token-kp44c: read tcp 172.16.0.23:36384->10.0.96.108:6443: use of closed network connection
>  Warning  FailedMount  24m                 kubelet, host-172-16-0-23  MountVolume.SetUp failed for volume "kubevirt-operator-token-kp44c" : Get https://api-int.working.oc4:6443/api/v1/namespaces/openshift-cnv/secrets/kubevirt-operator-token-kp44c: read tcp 172.16.0.23:36504->10.0.96.108:6443: use of closed network connection
>  Warning  FailedMount  24m                 kubelet, host-172-16-0-23  MountVolume.SetUp failed for volume "kubevirt-operator-token-kp44c" : Get https://api-int.working.oc4:6443/api/v1/namespaces/openshift-cnv/secrets/kubevirt-operator-token-kp44c: read tcp 172.16.0.23:36618->10.0.96.108:6443: use of closed network connection
>  Warning  FailedMount  23m                 kubelet, host-172-16-0-23  MountVolume.SetUp failed for volume "kubevirt-operator-token-kp44c" : Get https://api-int.working.oc4:6443/api/v1/namespaces/openshift-cnv/secrets/kubevirt-operator-token-kp44c: read tcp 172.16.0.23:36738->10.0.96.108:6443: use of closed network connection
>  Warning  FailedMount  23m                 kubelet, host-172-16-0-23  MountVolume.SetUp failed for volume "kubevirt-operator-token-kp44c" : Get https://api-int.working.oc4:6443/api/v1/namespaces/openshift-cnv/secrets/kubevirt-operator-token-kp44c: read tcp 172.16.0.23:37122->10.0.96.108:6443: use of closed network connection
>  Normal   Pulled       18m (x43 over 17h)  kubelet, host-172-16-0-23  Container image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/virt-operator:v2.1.0-7" already present on machine
>  Normal   Started      18m (x43 over 17h)  kubelet, host-172-16-0-23  Started container virt-operator
>  Normal   Created      18m (x43 over 17h)  kubelet, host-172-16-0-23  Created container virt-operator
>  Warning  Unhealthy    8m (x17 over 13h)   kubelet, host-172-16-0-23  Readiness probe failed: Get https://10.128.0.57:8443/metrics: dial tcp 10.128.0.57:8443: connect: connection refused
>  Warning  BackOff      2m (x181 over 13h)  kubelet, host-172-16-0-23  Back-off restarting failed container
>[kbidarka@kbidarka-host osdc]$ oc describe pod virt-operator-6c65f86c4f-jhcsj -n openshift-cnv
>Name:               virt-operator-6c65f86c4f-jhcsj
>Namespace:          openshift-cnv
>Priority:           2000000000
>PriorityClassName:  system-cluster-critical
>Node:               host-172-16-0-22/172.16.0.22
>Start Time:         Wed, 21 Aug 2019 22:35:07 +0530
>Labels:             kubevirt.io=virt-operator
>                    pod-template-hash=6c65f86c4f
>                    prometheus.kubevirt.io=
>Annotations:        alm-examples=[
>  {
>    "apiVersion": "hco.kubevirt.io/v1alpha1",
>    "kind": "HyperConverged",
>    "metadata": {
>      "name": "kubevirt-hyperconverged",
>      "namespace": "kubevirt-hyperconverged"
> ...
>                capabilities=Full Lifecycle
>                categories=OpenShift Optional
>                certified=false
>                containerImage=brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/container-native-virtualization/hyperconverged-cluster-operator:v2.1.0-6
>                createdAt=2019-08-21 15:44:09.993271
>                description=Creates and maintains a HyperConverged KubeVirt Deployment
>                olm.operatorGroup=hco-operatorgroup
>                olm.operatorNamespace=openshift-cnv
>                olm.targetNamespaces=
>                openshift.io/scc=restricted
>                repository=https://github.com/kubevirt/hyperconverged-cluster-operator
>                scheduler.alpha.kubernetes.io/critical-pod=
>                scheduler.alpha.kubernetes.io/tolerations=[{"key":"CriticalAddonsOnly","operator":"Exists"}]
>                support=false
>Status:         Running
>IP:             10.130.0.55
>Controlled By:  ReplicaSet/virt-operator-6c65f86c4f
>Containers:
>  virt-operator:
>    Container ID:  cri-o://cb93c7e6d53fbd2725e1e43ca9b0162e0b43814f60ad9fc4f8cca818f05ba06d
>    Image:         brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/virt-operator:v2.1.0-7
>    Image ID:      brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/virt-operator@sha256:7fc9386a6a307b60b190c866f84a375a05ee3b26b4b0eeb5867c1673a28c708b
>    Port:          8443/TCP
>    Host Port:     0/TCP
>    Command:
>      virt-operator
>      --port
>      8443
>      -v
>      2
>    State:          Waiting
>      Reason:       CrashLoopBackOff
>    Last State:     Terminated
>      Reason:       Error
>      Exit Code:    1
>      Started:      Thu, 22 Aug 2019 16:05:59 +0530
>      Finished:     Thu, 22 Aug 2019 16:10:45 +0530
>    Ready:          False
>    Restart Count:  56
>    Readiness:      http-get https://:8443/metrics delay=5s timeout=10s period=10s #success=1 #failure=3
>    Environment:
>      OPERATOR_IMAGE:   brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/virt-operator:v2.1.0-7
>      WATCH_NAMESPACE:   (v1:metadata.annotations['olm.targetNamespaces'])
>    Mounts:
>      /var/run/secrets/kubernetes.io/serviceaccount from kubevirt-operator-token-kp44c (ro)
>Conditions:
>  Type              Status
>  Initialized       True 
>  Ready             False 
>  ContainersReady   False 
>  PodScheduled      True 
>Volumes:
>  kubevirt-operator-token-kp44c:
>    Type:        Secret (a volume populated by a Secret)
>    SecretName:  kubevirt-operator-token-kp44c
>    Optional:    false
>QoS Class:       BestEffort
>Node-Selectors:  <none>
>Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
>                 node.kubernetes.io/unreachable:NoExecute for 300s
>Events:
>  Type     Reason       Age                 From                       Message
>  ----     ------       ----                ----                       -------
>  Warning  FailedMount  2h                  kubelet, host-172-16-0-22  MountVolume.SetUp failed for volume "kubevirt-operator-token-kp44c" : Get https://api-int.working.oc4:6443/api/v1/namespaces/openshift-cnv/secrets/kubevirt-operator-token-kp44c: read tcp 172.16.0.22:39182->10.0.96.108:6443: use of closed network connection
>  Warning  FailedMount  2h                  kubelet, host-172-16-0-22  MountVolume.SetUp failed for volume "kubevirt-operator-token-kp44c" : Get https://api-int.working.oc4:6443/api/v1/namespaces/openshift-cnv/secrets/kubevirt-operator-token-kp44c: read tcp 172.16.0.22:40358->10.0.96.108:6443: use of closed network connection
>  Warning  FailedMount  2h (x11 over 9h)    kubelet, host-172-16-0-22  MountVolume.SetUp failed for volume "kubevirt-operator-token-kp44c" : secrets "kubevirt-operator-token-kp44c" is forbidden: User "system:node:host-172-16-0-22" cannot get resource "secrets" in API group "" in the namespace "openshift-cnv": no relationship found between node "host-172-16-0-22" and this object
>  Warning  FailedMount  2h                  kubelet, host-172-16-0-22  MountVolume.SetUp failed for volume "kubevirt-operator-token-kp44c" : Get https://api-int.working.oc4:6443/api/v1/namespaces/openshift-cnv/secrets/kubevirt-operator-token-kp44c: read tcp 172.16.0.22:40900->10.0.96.108:6443: use of closed network connection
>  Warning  FailedMount  2h                  kubelet, host-172-16-0-22  MountVolume.SetUp failed for volume "kubevirt-operator-token-kp44c" : Get https://api-int.working.oc4:6443/api/v1/namespaces/openshift-cnv/secrets/kubevirt-operator-token-kp44c: read tcp 172.16.0.22:49324->10.0.96.108:6443: use of closed network connection
>  Warning  FailedMount  2h                  kubelet, host-172-16-0-22  MountVolume.SetUp failed for volume "kubevirt-operator-token-kp44c" : Get https://api-int.working.oc4:6443/api/v1/namespaces/openshift-cnv/secrets/kubevirt-operator-token-kp44c: read tcp 172.16.0.22:49352->10.0.96.108:6443: use of closed network connection
>  Warning  FailedMount  2h                  kubelet, host-172-16-0-22  MountVolume.SetUp failed for volume "kubevirt-operator-token-kp44c" : Get https://api-int.working.oc4:6443/api/v1/namespaces/openshift-cnv/secrets/kubevirt-operator-token-kp44c: read tcp 172.16.0.22:51398->10.0.96.108:6443: use of closed network connection
>  Warning  Unhealthy    23m (x7 over 13h)   kubelet, host-172-16-0-22  Readiness probe failed: Get https://10.130.0.55:8443/metrics: dial tcp 10.130.0.55:8443: connect: connection refused
>  Normal   Pulled       17m (x55 over 17h)  kubelet, host-172-16-0-22  Container image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/virt-operator:v2.1.0-7" already present on machine
>  Normal   Created      17m (x55 over 17h)  kubelet, host-172-16-0-22  Created container virt-operator
>  Normal   Started      17m (x55 over 17h)  kubelet, host-172-16-0-22  Started container virt-operator
>  Warning  BackOff      3m (x202 over 15h)  kubelet, host-172-16-0-22  Back-off restarting failed container
>[kbidarka@kbidarka-host osdc]$ oc describe pod virt-controller-dfb67f66-7pdpq -n openshift-cnv 
>Name:                      virt-controller-dfb67f66-7pdpq
>Namespace:                 openshift-cnv
>Priority:                  2000000000
>PriorityClassName:         system-cluster-critical
>Node:                      host-172-16-0-22/172.16.0.22
>Start Time:                Thu, 22 Aug 2019 16:04:41 +0530
>Labels:                    app.kubernetes.io/managed-by=kubevirt-operator
>                           kubevirt.io=virt-controller
>                           pod-template-hash=dfb67f66
>                           prometheus.kubevirt.io=
>Annotations:               kubevirt.io/install-strategy-identifier=995973abf6b8c7b406be8bb049a20aaf4f72eaeb
>                           kubevirt.io/install-strategy-registry=brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization
>                           kubevirt.io/install-strategy-version=v2.1.0-7
>                           openshift.io/scc=restricted
>                           scheduler.alpha.kubernetes.io/critical-pod=
>                           scheduler.alpha.kubernetes.io/tolerations=[{"key":"CriticalAddonsOnly","operator":"Exists"}]
>Status:                    Terminating (lasts 58s)
>Termination Grace Period:  30s
>IP:                        10.130.0.222
>Controlled By:             ReplicaSet/virt-controller-dfb67f66
>Containers:
>  virt-controller:
>    Container ID:  cri-o://e6ae7e13319080f07cce3e81d5a3ad991958ebe185143dc9a5b61266721d10e3
>    Image:         brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/virt-controller:v2.1.0-7
>    Image ID:      brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/virt-controller@sha256:33aa6c8487213d37dc02d44417ac3c0c65790867efd5f22898b28c6ecc408b59
>    Port:          8443/TCP
>    Host Port:     0/TCP
>    Command:
>      virt-controller
>      --launcher-image
>      brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/virt-launcher:v2.1.0-7
>      --port
>      8443
>      -v
>      2
>    State:          Running
>      Started:      Thu, 22 Aug 2019 16:13:56 +0530
>    Last State:     Terminated
>      Reason:       Error
>      Exit Code:    1
>      Started:      Thu, 22 Aug 2019 16:04:43 +0530
>      Finished:     Thu, 22 Aug 2019 16:13:55 +0530
>    Ready:          True
>    Restart Count:  1
>    Liveness:       http-get https://:8443/healthz delay=15s timeout=10s period=10s #success=1 #failure=8
>    Readiness:      http-get https://:8443/leader delay=15s timeout=10s period=10s #success=1 #failure=3
>    Environment:    <none>
>    Mounts:
>      /var/run/secrets/kubernetes.io/serviceaccount from kubevirt-controller-token-vk5pq (ro)
>Conditions:
>  Type              Status
>  Initialized       True 
>  Ready             True 
>  ContainersReady   True 
>  PodScheduled      True 
>Volumes:
>  kubevirt-controller-token-vk5pq:
>    Type:        Secret (a volume populated by a Secret)
>    SecretName:  kubevirt-controller-token-vk5pq
>    Optional:    false
>QoS Class:       BestEffort
>Node-Selectors:  <none>
>Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
>                 node.kubernetes.io/unreachable:NoExecute for 300s
>Events:
>  Type     Reason            Age               From                       Message
>  ----     ------            ----              ----                       -------
>  Warning  FailedScheduling  13m               default-scheduler          Binding rejected: Operation cannot be fulfilled on pods/binding "virt-controller-dfb67f66-7pdpq": pod virt-controller-dfb67f66-7pdpq is already assigned to node "host-172-16-0-22"
>  Normal   Scheduled         13m               default-scheduler          Successfully assigned openshift-cnv/virt-controller-dfb67f66-7pdpq to host-172-16-0-22
>  Warning  FailedScheduling  13m               default-scheduler          Binding rejected: Operation cannot be fulfilled on pods/binding "virt-controller-dfb67f66-7pdpq": pod virt-controller-dfb67f66-7pdpq is already assigned to node "host-172-16-0-22"
>  Warning  Unhealthy         3m                kubelet, host-172-16-0-22  Liveness probe failed: Get https://10.130.0.222:8443/healthz: unexpected EOF
>  Normal   Pulled            3m (x2 over 12m)  kubelet, host-172-16-0-22  Container image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/container-native-virtualization/virt-controller:v2.1.0-7" already present on machine
>  Normal   Created           3m (x2 over 12m)  kubelet, host-172-16-0-22  Created container virt-controller
>  Normal   Started           3m (x2 over 12m)  kubelet, host-172-16-0-22  Started container virt-controller
>  Normal   Killing           1m                kubelet, host-172-16-0-22  Stopping container virt-controller
>

Actions: View

Attachments on bug 1744534: 1606916