Bug 1595395 - VM doesn't stay running - "Still missing PID for qemu-system, Process qemu-system not found in /proc"
Summary: VM doesn't stay running - "Still missing PID for qemu-system, Process qemu-sy...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Container Native Virtualization (CNV)
Classification: Red Hat
Component: Virtualization
Version: 1.1
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 1.1
Assignee: David Vossel
QA Contact: Gabriel Szasz
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-06-26 19:59 UTC by Steve Reichard
Modified: 2018-11-05 13:49 UTC (History)
5 users (show)

Fixed In Version: kubevirt-0.6.2
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-11-05 13:49:30 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Steve Reichard 2018-06-26 19:59:50 UTC
Description of problem:

After installing the downstream CNV and install by creating the kubevirt manifest,  I attempted to launch a VM from:  https://raw.githubusercontent.com/kubevirt/demo/master/manifests/vm.yaml


After 5-6 minutes the VM is killed and is restated:

While I was able to launch a VM that showed ready, it didn't stay running after 5 minutes.

Below is a log from the compute container,  I've truncated the last message which repeats several times.

I see domain started, sync'd VMI,  any assistance as to what may be holding up the PID?

[root@ospha1 ~]# oc logs po/virt-launcher-testvm-ksr4b -c compute
level=info timestamp=2018-06-25T20:16:59.209125Z pos=libvirt.go:256 component=virt-launcher msg="Connecting to libvirt daemon: qemu:///system"
level=info timestamp=2018-06-25T20:17:09.212857Z pos=libvirt.go:271 component=virt-launcher msg="Connected to libvirt daemon"
level=info timestamp=2018-06-25T20:17:09.238320Z pos=virt-launcher.go:142 component=virt-launcher msg="Watchdog file created at /var/run/kubevirt/watchdog-files/kube-system_testvm"
level=info timestamp=2018-06-25T20:17:09.239065Z pos=client.go:152 component=virt-launcher msg="Registered libvirt event notify callback"
level=info timestamp=2018-06-25T20:17:09.239323Z pos=virt-launcher.go:59 component=virt-launcher msg="Marked as ready"
level=info timestamp=2018-06-25T20:17:09.239646Z pos=monitor.go:253 component=virt-launcher msg="Monitoring loop: rate 1s start timeout 5m0s"
level=info timestamp=2018-06-25T20:17:10.243675Z pos=monitor.go:209 component=virt-launcher msg="Still missing PID for qemu-system, Process qemu-system not found in /proc"
level=info timestamp=2018-06-25T20:17:11.218645Z pos=cloud-init.go:254 component=virt-launcher msg="generated nocloud iso file /var/run/libvirt/kubevirt-ephemeral-disk/cloud-init-data/kube-system/testvm/noCloud.iso"
level=error timestamp=2018-06-25T20:17:11.240161Z pos=common.go:130 component=virt-launcher msg="updated MAC for interface: eth0 - 0a:58:0a:20:26:9d"
level=info timestamp=2018-06-25T20:17:11.242507Z pos=common.go:245 component=virt-launcher msg="Found nameservers in /etc/resolv.conf: \n\u0013\ufffd#"
level=info timestamp=2018-06-25T20:17:11.242573Z pos=common.go:246 component=virt-launcher msg="Found search domains in /etc/resolv.conf: kube-system.svc.cluster.local svc.cluster.local cluster.local cloud.lab.eng.bos.redhat.com"
level=info timestamp=2018-06-25T20:17:11.242735Z pos=dhcp.go:62 component=virt-launcher msg="Starting SingleClientDHCPServer"
level=info timestamp=2018-06-25T20:17:11.243009Z pos=monitor.go:209 component=virt-launcher msg="Still missing PID for qemu-system, Process qemu-system not found in /proc"
level=info timestamp=2018-06-25T20:17:11.402088Z pos=manager.go:151 component=virt-launcher namespace=kube-system name=testvm kind= uid=a787b688-78b4-11e8-ab46-e0db5574412f msg="Domain defined."
level=info timestamp=2018-06-25T20:17:11.402541Z pos=client.go:136 component=virt-launcher msg="Libvirt event 0 with reason 0 received"
level=info timestamp=2018-06-25T20:17:12.241867Z pos=monitor.go:209 component=virt-launcher msg="Still missing PID for qemu-system, Process qemu-system not found in /proc"
level=info timestamp=2018-06-25T20:17:12.283708Z pos=client.go:119 component=virt-launcher msg="domain status: 3:11"
level=info timestamp=2018-06-25T20:17:12.292297Z pos=client.go:145 component=virt-launcher msg="processed event"
level=info timestamp=2018-06-25T20:17:12.505202Z pos=client.go:136 component=virt-launcher msg="Libvirt event 4 with reason 0 received"
level=info timestamp=2018-06-25T20:17:12.527637Z pos=client.go:119 component=virt-launcher msg="domain status: 1:1"
level=info timestamp=2018-06-25T20:17:12.528779Z pos=client.go:145 component=virt-launcher msg="processed event"
level=info timestamp=2018-06-25T20:17:12.541627Z pos=client.go:136 component=virt-launcher msg="Libvirt event 2 with reason 0 received"
level=info timestamp=2018-06-25T20:17:12.556480Z pos=manager.go:182 component=virt-launcher namespace=kube-system name=testvm kind= uid=a787b688-78b4-11e8-ab46-e0db5574412f msg="Domain started."
level=info timestamp=2018-06-25T20:17:12.557413Z pos=server.go:74 component=virt-launcher namespace=kube-system name=testvm kind= uid=a787b688-78b4-11e8-ab46-e0db5574412f msg="Synced vmi"
level=info timestamp=2018-06-25T20:17:12.557746Z pos=client.go:119 component=virt-launcher msg="domain status: 1:1"
level=info timestamp=2018-06-25T20:17:12.559944Z pos=client.go:145 component=virt-launcher msg="processed event"
level=info timestamp=2018-06-25T20:17:12.589727Z pos=server.go:74 component=virt-launcher namespace=kube-system name=testvm kind= uid=a787b688-78b4-11e8-ab46-e0db5574412f msg="Synced vmi"
level=info timestamp=2018-06-25T20:17:12.594620Z pos=server.go:74 component=virt-launcher namespace=kube-system name=testvm kind= uid=a787b688-78b4-11e8-ab46-e0db5574412f msg="Synced vmi"
level=info timestamp=2018-06-25T20:17:13.243537Z pos=monitor.go:209 component=virt-launcher msg="Still missing PID for qemu-system, Process qemu-system not found in /proc"
level=info timestamp=2018-06-25T20:17:14.241672Z pos=monitor.go:209 component=virt-launcher msg="Still missing PID for qemu-system, Process qemu-system not found in /proc"
[truncated]



Event output:

[root@ospha1 ~]# oc get events
LAST SEEN   FIRST SEEN   COUNT     NAME                                                KIND                     SUBOBJECT                               TYPE      REASON               SOURCE                                              MESSAGE
9m          9m           1         testvm.153b800cdd1c4cff                             VirtualMachineInstance                                           Normal    Deleted              virt-handler, ospha3.cloud.lab.eng.bos.redhat.com   VirtualMachineInstance stopping
9m          9m           1         testvm.153b800cdfe8023f                             VirtualMachineInstance                                           Warning   Stopped              virt-handler, ospha3.cloud.lab.eng.bos.redhat.com   The VirtualMachineInstance crashed.
41s         6m           2         testvm.153b803177e1bb52                             VirtualMachine                                                   Normal    SuccessfulCreate     virtualmachine-controller                           Created virtual machine: testvm
6m          6m           1         testvm.153b80317991a900                             VirtualMachineInstance                                           Normal    SuccessfulCreate     virtualmachine-controller                           Created virtual machine pod virt-launcher-testvm-mmv5f
5m          5m           1         testvm.153b8039f8bca34f                             VirtualMachineInstance                                           Normal    SuccessfulHandOver   virtualmachine-controller                           Pod owner ship transferred to the node virt-launcher-testvm-mmv5f
5m          5m           2         testvm.153b803a4cc6d735                             VirtualMachineInstance                                           Normal    Created              virt-handler, ospha2.cloud.lab.eng.bos.redhat.com   VirtualMachineInstance defined.
5m          5m           1         testvm.153b803a4f7a6c63                             VirtualMachineInstance                                           Normal    Started              virt-handler, ospha2.cloud.lab.eng.bos.redhat.com   VirtualMachineInstance started.
51s         51s          1         testvm.153b807f7d8d0d2e                             VirtualMachineInstance                                           Warning   Stopped              virt-handler, ospha2.cloud.lab.eng.bos.redhat.com   The VirtualMachineInstance crashed.
51s         51s          2         testvm.153b807f7e0dd48a                             VirtualMachine                                                   Normal    SuccessfulDelete     virtualmachine-controller                           Deleted virtual machine: d9d101ed-78b3-11e8-ab46-e0db5574412f
51s         51s          1         testvm.153b807f7edcb045                             VirtualMachineInstance                                           Normal    SuccessfulDelete     virtualmachine-controller                           Deleted virtual machine pod virt-launcher-testvm-mmv5f
41s         41s          1         testvm.153b8081d4a41781                             VirtualMachineInstance                                           Normal    SuccessfulCreate     virtualmachine-controller                           Created virtual machine pod virt-launcher-testvm-ksr4b
3s          3s           1         testvm.153b808a98c5b1ba                             VirtualMachineInstance                                           Normal    SuccessfulHandOver   virtualmachine-controller                           Pod owner ship transferred to the node virt-launcher-testvm-ksr4b
2s          2s           3         testvm.153b808aeb05bfec                             VirtualMachineInstance                                           Normal    Created              virt-handler, ospha5.cloud.lab.eng.bos.redhat.com   VirtualMachineInstance defined.
2s          2s           1         testvm.153b808aec72f616                             VirtualMachineInstance                                           Normal    Started              virt-handler, ospha5.cloud.lab.eng.bos.redhat.com   VirtualMachineInstance started.
9m          9m           1         virt-api-7bc9bcb7bb-6dwsc.153b800c38168b27          Pod                                                              Normal    Scheduled            default-scheduler                                   Successfully assigned virt-api-7bc9bcb7bb-6dwsc to ospha5.cloud.lab.eng.bos.redhat.com
9m          9m           1         virt-api-7bc9bcb7bb-6dwsc.153b800ce73bae38          Pod                      spec.containers{virt-api}               Normal    Pulled               kubelet, ospha5.cloud.lab.eng.bos.redhat.com        Container image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/cnv11-tech-preview/virt-api:1.1" already present on machine
9m          9m           1         virt-api-7bc9bcb7bb-6dwsc.153b800cedbc7c24          Pod                      spec.containers{virt-api}               Normal    Created              kubelet, ospha5.cloud.lab.eng.bos.redhat.com        Created container
9m          9m           1         virt-api-7bc9bcb7bb-6dwsc.153b800cfb9e0a59          Pod                      spec.containers{virt-api}               Normal    Started              kubelet, ospha5.cloud.lab.eng.bos.redhat.com        Started container
8m          8m           1         virt-api-7bc9bcb7bb-6dwsc.153b80157b01d8b2          Pod                      spec.containers{virt-api}               Normal    Killing              kubelet, ospha5.cloud.lab.eng.bos.redhat.com        Killing container with id docker://virt-api:Need to kill Pod
7m          7m           1         virt-api-7bc9bcb7bb-7t7qm.153b801dcab55e33          Pod                                                              Normal    Scheduled            default-scheduler                                   Successfully assigned virt-api-7bc9bcb7bb-7t7qm to ospha5.cloud.lab.eng.bos.redhat.com
7m          7m           1         virt-api-7bc9bcb7bb-7t7qm.153b801e502601ff          Pod                      spec.containers{virt-api}               Normal    Pulled               kubelet, ospha5.cloud.lab.eng.bos.redhat.com        Container image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/cnv11-tech-preview/virt-api:1.1" already present on machine
7m          7m           1         virt-api-7bc9bcb7bb-7t7qm.153b801e55fcfc0f          Pod                      spec.containers{virt-api}               Normal    Created              kubelet, ospha5.cloud.lab.eng.bos.redhat.com        Created container
7m          7m           1         virt-api-7bc9bcb7bb-7t7qm.153b801e61b48740          Pod                      spec.containers{virt-api}               Normal    Started              kubelet, ospha5.cloud.lab.eng.bos.redhat.com        Started container
9m          9m           1         virt-api-7bc9bcb7bb-8zxkb.153b800c370210a9          Pod                                                              Normal    Scheduled            default-scheduler                                   Successfully assigned virt-api-7bc9bcb7bb-8zxkb to ospha2.cloud.lab.eng.bos.redhat.com
9m          9m           1         virt-api-7bc9bcb7bb-8zxkb.153b800cd5f2e5cc          Pod                      spec.containers{virt-api}               Normal    Pulled               kubelet, ospha2.cloud.lab.eng.bos.redhat.com        Container image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/cnv11-tech-preview/virt-api:1.1" already present on machine
9m          9m           1         virt-api-7bc9bcb7bb-8zxkb.153b800cdc8a91d9          Pod                      spec.containers{virt-api}               Normal    Created              kubelet, ospha2.cloud.lab.eng.bos.redhat.com        Created container
9m          9m           1         virt-api-7bc9bcb7bb-8zxkb.153b800ce9280002          Pod                      spec.containers{virt-api}               Normal    Started              kubelet, ospha2.cloud.lab.eng.bos.redhat.com        Started container
8m          8m           1         virt-api-7bc9bcb7bb-8zxkb.153b80157a77b551          Pod                      spec.containers{virt-api}               Normal    Killing              kubelet, ospha2.cloud.lab.eng.bos.redhat.com        Killing container with id docker://virt-api:Need to kill Pod
7m          7m           1         virt-api-7bc9bcb7bb-d2g2k.153b801dcab0ea92          Pod                                                              Normal    Scheduled            default-scheduler                                   Successfully assigned virt-api-7bc9bcb7bb-d2g2k to ospha3.cloud.lab.eng.bos.redhat.com
7m          7m           1         virt-api-7bc9bcb7bb-d2g2k.153b801e52e717eb          Pod                      spec.containers{virt-api}               Normal    Pulling              kubelet, ospha3.cloud.lab.eng.bos.redhat.com        pulling image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/cnv11-tech-preview/virt-api:1.1"
7m          7m           1         virt-api-7bc9bcb7bb-d2g2k.153b801fa1337751          Pod                      spec.containers{virt-api}               Normal    Pulled               kubelet, ospha3.cloud.lab.eng.bos.redhat.com        Successfully pulled image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/cnv11-tech-preview/virt-api:1.1"
7m          7m           1         virt-api-7bc9bcb7bb-d2g2k.153b801fa7b5ab39          Pod                      spec.containers{virt-api}               Normal    Created              kubelet, ospha3.cloud.lab.eng.bos.redhat.com        Created container
7m          7m           1         virt-api-7bc9bcb7bb-d2g2k.153b801fb3b40c6f          Pod                      spec.containers{virt-api}               Normal    Started              kubelet, ospha3.cloud.lab.eng.bos.redhat.com        Started container
9m          9m           1         virt-api-7bc9bcb7bb.153b800c361db958                ReplicaSet                                                       Normal    SuccessfulCreate     replicaset-controller                               Created pod: virt-api-7bc9bcb7bb-8zxkb
9m          9m           1         virt-api-7bc9bcb7bb.153b800c36fbdeb5                ReplicaSet                                                       Normal    SuccessfulCreate     replicaset-controller                               Created pod: virt-api-7bc9bcb7bb-6dwsc
8m          8m           1         virt-api-7bc9bcb7bb.153b801572e89c71                ReplicaSet                                                       Normal    SuccessfulDelete     replicaset-controller                               Deleted pod: virt-api-7bc9bcb7bb-6dwsc
8m          8m           1         virt-api-7bc9bcb7bb.153b8015738bc5c3                ReplicaSet                                                       Normal    SuccessfulDelete     replicaset-controller                               Deleted pod: virt-api-7bc9bcb7bb-8zxkb
7m          7m           1         virt-api-7bc9bcb7bb.153b801dc98a93a1                ReplicaSet                                                       Normal    SuccessfulCreate     replicaset-controller                               Created pod: virt-api-7bc9bcb7bb-7t7qm
7m          7m           1         virt-api-7bc9bcb7bb.153b801dca122efc                ReplicaSet                                                       Normal    SuccessfulCreate     replicaset-controller                               Created pod: virt-api-7bc9bcb7bb-d2g2k
9m          9m           1         virt-api.153b800c2ecaaf68                           Deployment                                                       Normal    ScalingReplicaSet    deployment-controller                               Scaled up replica set virt-api-7bc9bcb7bb to 2
8m          8m           1         virt-api.153b80157061f3f6                           Deployment                                                       Normal    ScalingReplicaSet    deployment-controller                               Scaled down replica set virt-api-7bc9bcb7bb to 0
7m          7m           1         virt-api.153b801dc6ef1be4                           Deployment                                                       Normal    ScalingReplicaSet    deployment-controller                               Scaled up replica set virt-api-7bc9bcb7bb to 2
9m          9m           1         virt-controller-644b687978-d5wrl.153b800c38156f23   Pod                                                              Normal    Scheduled            default-scheduler                                   Successfully assigned virt-controller-644b687978-d5wrl to ospha4.cloud.lab.eng.bos.redhat.com
9m          9m           1         virt-controller-644b687978-d5wrl.153b800ce431c796   Pod                      spec.containers{virt-controller}        Normal    Pulling              kubelet, ospha4.cloud.lab.eng.bos.redhat.com        pulling image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/cnv11-tech-preview/virt-controller:1.1"
8m          8m           1         virt-controller-644b687978-d5wrl.153b800e0da03f22   Pod                      spec.containers{virt-controller}        Normal    Pulled               kubelet, ospha4.cloud.lab.eng.bos.redhat.com        Successfully pulled image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/cnv11-tech-preview/virt-controller:1.1"
8m          8m           1         virt-controller-644b687978-d5wrl.153b800e15369c3f   Pod                      spec.containers{virt-controller}        Normal    Created              kubelet, ospha4.cloud.lab.eng.bos.redhat.com        Created container
8m          8m           1         virt-controller-644b687978-d5wrl.153b800e215c35d0   Pod                      spec.containers{virt-controller}        Normal    Started              kubelet, ospha4.cloud.lab.eng.bos.redhat.com        Started container
8m          8m           1         virt-controller-644b687978-d5wrl.153b8015bb960677   Pod                      spec.containers{virt-controller}        Normal    Killing              kubelet, ospha4.cloud.lab.eng.bos.redhat.com        Killing container with id docker://virt-controller:Need to kill Pod
7m          7m           1         virt-controller-644b687978-dwn2d.153b801dcc7bdde1   Pod                                                              Normal    Scheduled            default-scheduler                                   Successfully assigned virt-controller-644b687978-dwn2d to ospha4.cloud.lab.eng.bos.redhat.com
7m          7m           1         virt-controller-644b687978-dwn2d.153b801e4e43a6e0   Pod                      spec.containers{virt-controller}        Normal    Pulled               kubelet, ospha4.cloud.lab.eng.bos.redhat.com        Container image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/cnv11-tech-preview/virt-controller:1.1" already present on machine
7m          7m           1         virt-controller-644b687978-dwn2d.153b801e5c852b49   Pod                      spec.containers{virt-controller}        Normal    Created              kubelet, ospha4.cloud.lab.eng.bos.redhat.com        Created container
7m          7m           1         virt-controller-644b687978-dwn2d.153b801e68f1b8ba   Pod                      spec.containers{virt-controller}        Normal    Started              kubelet, ospha4.cloud.lab.eng.bos.redhat.com        Started container
7m          7m           1         virt-controller-644b687978-rqfx4.153b801dcaa5ab64   Pod                                                              Normal    Scheduled            default-scheduler                                   Successfully assigned virt-controller-644b687978-rqfx4 to ospha2.cloud.lab.eng.bos.redhat.com
7m          7m           1         virt-controller-644b687978-rqfx4.153b801e5981d21f   Pod                      spec.containers{virt-controller}        Normal    Pulled               kubelet, ospha2.cloud.lab.eng.bos.redhat.com        Container image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/cnv11-tech-preview/virt-controller:1.1" already present on machine
7m          7m           1         virt-controller-644b687978-rqfx4.153b801e602ff5c6   Pod                      spec.containers{virt-controller}        Normal    Created              kubelet, ospha2.cloud.lab.eng.bos.redhat.com        Created container
7m          7m           1         virt-controller-644b687978-rqfx4.153b801e6cd99ebc   Pod                      spec.containers{virt-controller}        Normal    Started              kubelet, ospha2.cloud.lab.eng.bos.redhat.com        Started container
9m          9m           1         virt-controller-644b687978-vrfc5.153b800c39896d65   Pod                                                              Normal    Scheduled            default-scheduler                                   Successfully assigned virt-controller-644b687978-vrfc5 to ospha5.cloud.lab.eng.bos.redhat.com
9m          9m           1         virt-controller-644b687978-vrfc5.153b800cf0712bf2   Pod                      spec.containers{virt-controller}        Normal    Pulled               kubelet, ospha5.cloud.lab.eng.bos.redhat.com        Container image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/cnv11-tech-preview/virt-controller:1.1" already present on machine
9m          9m           1         virt-controller-644b687978-vrfc5.153b800cf70af596   Pod                      spec.containers{virt-controller}        Normal    Created              kubelet, ospha5.cloud.lab.eng.bos.redhat.com        Created container
9m          9m           1         virt-controller-644b687978-vrfc5.153b800d03c82bd4   Pod                      spec.containers{virt-controller}        Normal    Started              kubelet, ospha5.cloud.lab.eng.bos.redhat.com        Started container
8m          8m           1         virt-controller-644b687978-vrfc5.153b8015bb2cc67c   Pod                      spec.containers{virt-controller}        Normal    Killing              kubelet, ospha5.cloud.lab.eng.bos.redhat.com        Killing container with id docker://virt-controller:Need to kill Pod
9m          9m           1         virt-controller-644b687978.153b800c37075266         ReplicaSet                                                       Normal    SuccessfulCreate     replicaset-controller                               Created pod: virt-controller-644b687978-d5wrl
9m          9m           1         virt-controller-644b687978.153b800c38089c34         ReplicaSet                                                       Normal    SuccessfulCreate     replicaset-controller                               Created pod: virt-controller-644b687978-vrfc5
8m          8m           1         virt-controller-644b687978.153b8015b3f8808e         ReplicaSet                                                       Normal    SuccessfulDelete     replicaset-controller                               Deleted pod: virt-controller-644b687978-d5wrl
8m          8m           1         virt-controller-644b687978.153b8015b3f9eb62         ReplicaSet                                                       Normal    SuccessfulDelete     replicaset-controller                               Deleted pod: virt-controller-644b687978-vrfc5
7m          7m           1         virt-controller-644b687978.153b801dc98eb137         ReplicaSet                                                       Normal    SuccessfulCreate     replicaset-controller                               Created pod: virt-controller-644b687978-rqfx4
7m          7m           1         virt-controller-644b687978.153b801dca0deb17         ReplicaSet                                                       Normal    SuccessfulCreate     replicaset-controller                               Created pod: virt-controller-644b687978-dwn2d
9m          9m           1         virt-controller.153b800c2eafba78                    Deployment                                                       Normal    ScalingReplicaSet    deployment-controller                               Scaled up replica set virt-controller-644b687978 to 2
8m          8m           1         virt-controller.153b8011274106ec                    Endpoints                                                        Normal    LeaderElection       virt-controller                                     virt-controller-644b687978-vrfc5 became leader
8m          8m           1         virt-controller.153b8015b3810beb                    Deployment                                                       Normal    ScalingReplicaSet    deployment-controller                               Scaled down replica set virt-controller-644b687978 to 0
7m          7m           1         virt-controller.153b801dc7309ffb                    Deployment                                                       Normal    ScalingReplicaSet    deployment-controller                               Scaled up replica set virt-controller-644b687978 to 2
7m          7m           1         virt-controller.153b80228c868b8f                    Endpoints                                                        Normal    LeaderElection       virt-controller                                     virt-controller-644b687978-dwn2d became leader
7m          7m           1         virt-handler-2jnb7.153b801e4dda6159                 Pod                      spec.containers{virt-handler}           Normal    Pulled               kubelet, ospha4.cloud.lab.eng.bos.redhat.com        Container image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/cnv11-tech-preview/virt-handler:1.1" already present on machine
7m          7m           1         virt-handler-2jnb7.153b801e5bcf2b6a                 Pod                      spec.containers{virt-handler}           Normal    Created              kubelet, ospha4.cloud.lab.eng.bos.redhat.com        Created container
7m          7m           1         virt-handler-2jnb7.153b801e6a99e0a2                 Pod                      spec.containers{virt-handler}           Normal    Started              kubelet, ospha4.cloud.lab.eng.bos.redhat.com        Started container
9m          9m           1         virt-handler-5wx8k.153b800ce0ce7f9f                 Pod                      spec.containers{virt-handler}           Normal    Pulled               kubelet, ospha5.cloud.lab.eng.bos.redhat.com        Container image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/cnv11-tech-preview/virt-handler:1.1" already present on machine
9m          9m           1         virt-handler-5wx8k.153b800ce706ac6d                 Pod                      spec.containers{virt-handler}           Normal    Created              kubelet, ospha5.cloud.lab.eng.bos.redhat.com        Created container
9m          9m           1         virt-handler-5wx8k.153b800cf55ec911                 Pod                      spec.containers{virt-handler}           Normal    Started              kubelet, ospha5.cloud.lab.eng.bos.redhat.com        Started container
8m          8m           1         virt-handler-5wx8k.153b8015fc4e7205                 Pod                      spec.containers{virt-handler}           Normal    Killing              kubelet, ospha5.cloud.lab.eng.bos.redhat.com        Killing container with id docker://virt-handler:Need to kill Pod
9m          9m           1         virt-handler-9t4cv.153b800cce8708ea                 Pod                      spec.containers{virt-handler}           Normal    Pulled               kubelet, ospha2.cloud.lab.eng.bos.redhat.com        Container image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/cnv11-tech-preview/virt-handler:1.1" already present on machine
9m          9m           1         virt-handler-9t4cv.153b800cd60f51e4                 Pod                      spec.containers{virt-handler}           Normal    Created              kubelet, ospha2.cloud.lab.eng.bos.redhat.com        Created container
9m          9m           1         virt-handler-9t4cv.153b800ce46b7942                 Pod                      spec.containers{virt-handler}           Normal    Started              kubelet, ospha2.cloud.lab.eng.bos.redhat.com        Started container
8m          8m           1         virt-handler-9t4cv.153b8015fc9edd15                 Pod                      spec.containers{virt-handler}           Normal    Killing              kubelet, ospha2.cloud.lab.eng.bos.redhat.com        Killing container with id docker://virt-handler:Need to kill Pod
7m          7m           1         virt-handler-fqmq6.153b801e45a5082a                 Pod                      spec.containers{virt-handler}           Normal    Pulled               kubelet, ospha3.cloud.lab.eng.bos.redhat.com        Container image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/cnv11-tech-preview/virt-handler:1.1" already present on machine
7m          7m           1         virt-handler-fqmq6.153b801e4fb2ec52                 Pod                      spec.containers{virt-handler}           Normal    Created              kubelet, ospha3.cloud.lab.eng.bos.redhat.com        Created container
7m          7m           1         virt-handler-fqmq6.153b801e5de2e5c2                 Pod                      spec.containers{virt-handler}           Normal    Started              kubelet, ospha3.cloud.lab.eng.bos.redhat.com        Started container
9m          9m           1         virt-handler-m7f8k.153b800cd5f14d32                 Pod                      spec.containers{virt-handler}           Normal    Pulled               kubelet, ospha4.cloud.lab.eng.bos.redhat.com        Container image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/cnv11-tech-preview/virt-handler:1.1" already present on machine
9m          9m           1         virt-handler-m7f8k.153b800cdc11be27                 Pod                      spec.containers{virt-handler}           Normal    Created              kubelet, ospha4.cloud.lab.eng.bos.redhat.com        Created container
9m          9m           1         virt-handler-m7f8k.153b800ce9ec5ca8                 Pod                      spec.containers{virt-handler}           Normal    Started              kubelet, ospha4.cloud.lab.eng.bos.redhat.com        Started container
8m          8m           1         virt-handler-m7f8k.153b8015fbfe0239                 Pod                      spec.containers{virt-handler}           Normal    Killing              kubelet, ospha4.cloud.lab.eng.bos.redhat.com        Killing container with id docker://virt-handler:Need to kill Pod
7m          7m           1         virt-handler-sbrdk.153b801e50981861                 Pod                      spec.containers{virt-handler}           Normal    Pulled               kubelet, ospha2.cloud.lab.eng.bos.redhat.com        Container image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/cnv11-tech-preview/virt-handler:1.1" already present on machine
7m          7m           1         virt-handler-sbrdk.153b801e590618d4                 Pod                      spec.containers{virt-handler}           Normal    Created              kubelet, ospha2.cloud.lab.eng.bos.redhat.com        Created container
7m          7m           1         virt-handler-sbrdk.153b801e6769170d                 Pod                      spec.containers{virt-handler}           Normal    Started              kubelet, ospha2.cloud.lab.eng.bos.redhat.com        Started container
9m          9m           1         virt-handler-ttznd.153b800cb2f9e267                 Pod                      spec.containers{virt-handler}           Normal    Pulled               kubelet, ospha3.cloud.lab.eng.bos.redhat.com        Container image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/cnv11-tech-preview/virt-handler:1.1" already present on machine
9m          9m           1         virt-handler-ttznd.153b800cbb6c493a                 Pod                      spec.containers{virt-handler}           Normal    Created              kubelet, ospha3.cloud.lab.eng.bos.redhat.com        Created container
9m          9m           1         virt-handler-ttznd.153b800cc9c825fd                 Pod                      spec.containers{virt-handler}           Normal    Started              kubelet, ospha3.cloud.lab.eng.bos.redhat.com        Started container
8m          8m           1         virt-handler-ttznd.153b8015fcba6725                 Pod                      spec.containers{virt-handler}           Normal    Killing              kubelet, ospha3.cloud.lab.eng.bos.redhat.com        Killing container with id docker://virt-handler:Need to kill Pod
7m          7m           1         virt-handler-wsnfp.153b801e715362b7                 Pod                      spec.containers{virt-handler}           Normal    Pulled               kubelet, ospha5.cloud.lab.eng.bos.redhat.com        Container image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/cnv11-tech-preview/virt-handler:1.1" already present on machine
7m          7m           1         virt-handler-wsnfp.153b801e7abda8e8                 Pod                      spec.containers{virt-handler}           Normal    Created              kubelet, ospha5.cloud.lab.eng.bos.redhat.com        Created container
7m          7m           1         virt-handler-wsnfp.153b801e88d5fd3f                 Pod                      spec.containers{virt-handler}           Normal    Started              kubelet, ospha5.cloud.lab.eng.bos.redhat.com        Started container
9m          9m           1         virt-handler.153b800c3067b204                       DaemonSet                                                        Normal    SuccessfulCreate     daemonset-controller                                Created pod: virt-handler-5wx8k
9m          9m           1         virt-handler.153b800c3265aa36                       DaemonSet                                                        Normal    SuccessfulCreate     daemonset-controller                                Created pod: virt-handler-ttznd
9m          9m           1         virt-handler.153b800c3265ac8e                       DaemonSet                                                        Normal    SuccessfulCreate     daemonset-controller                                Created pod: virt-handler-9t4cv
9m          9m           1         virt-handler.153b800c33a9b58b                       DaemonSet                                                        Normal    SuccessfulCreate     daemonset-controller                                Created pod: virt-handler-m7f8k
8m          8m           1         virt-handler.153b8015f4508bca                       DaemonSet                                                        Normal    SuccessfulDelete     daemonset-controller                                Deleted pod: virt-handler-5wx8k
8m          8m           1         virt-handler.153b8015f47fde4d                       DaemonSet                                                        Normal    SuccessfulDelete     daemonset-controller                                Deleted pod: virt-handler-9t4cv
8m          8m           1         virt-handler.153b8015f47feff9                       DaemonSet                                                        Normal    SuccessfulDelete     daemonset-controller                                Deleted pod: virt-handler-m7f8k
8m          8m           1         virt-handler.153b8015f4820d05                       DaemonSet                                                        Normal    SuccessfulDelete     daemonset-controller                                Deleted pod: virt-handler-ttznd
7m          7m           1         virt-handler.153b801dc89e87b1                       DaemonSet                                                        Normal    SuccessfulCreate     daemonset-controller                                Created pod: virt-handler-2jnb7
7m          7m           1         virt-handler.153b801dc90a755d                       DaemonSet                                                        Normal    SuccessfulCreate     daemonset-controller                                Created pod: virt-handler-wsnfp
7m          7m           1         virt-handler.153b801dc931fdc5                       DaemonSet                                                        Normal    SuccessfulCreate     daemonset-controller                                Created pod: virt-handler-fqmq6
7m          7m           1         virt-handler.153b801dca194583                       DaemonSet                                                        Normal    SuccessfulCreate     daemonset-controller                                Created pod: virt-handler-sbrdk
41s         41s          1         virt-launcher-testvm-ksr4b.153b8081d5f34689         Pod                                                              Normal    Scheduled            default-scheduler                                   Successfully assigned virt-launcher-testvm-ksr4b to ospha5.cloud.lab.eng.bos.redhat.com
38s         38s          1         virt-launcher-testvm-ksr4b.153b80827d464061         Pod                      spec.containers{volumeregistryvolume}   Normal    Pulled               kubelet, ospha5.cloud.lab.eng.bos.redhat.com        Container image "kubevirt/cirros-registry-disk-demo" already present on machine
38s         38s          1         virt-launcher-testvm-ksr4b.153b808284701de9         Pod                      spec.containers{volumeregistryvolume}   Normal    Created              kubelet, ospha5.cloud.lab.eng.bos.redhat.com        Created container
38s         38s          1         virt-launcher-testvm-ksr4b.153b808291712fe8         Pod                      spec.containers{volumeregistryvolume}   Normal    Started              kubelet, ospha5.cloud.lab.eng.bos.redhat.com        Started container
38s         38s          1         virt-launcher-testvm-ksr4b.153b808291aad385         Pod                      spec.containers{compute}                Normal    Pulling              kubelet, ospha5.cloud.lab.eng.bos.redhat.com        pulling image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/cnv11-tech-preview/virt-launcher:1.1"
16s         16s          1         virt-launcher-testvm-ksr4b.153b80879ab5e680         Pod                      spec.containers{compute}                Normal    Pulled               kubelet, ospha5.cloud.lab.eng.bos.redhat.com        Successfully pulled image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/cnv11-tech-preview/virt-launcher:1.1"
16s         16s          1         virt-launcher-testvm-ksr4b.153b8087a179b13f         Pod                      spec.containers{compute}                Normal    Created              kubelet, ospha5.cloud.lab.eng.bos.redhat.com        Created container
16s         16s          1         virt-launcher-testvm-ksr4b.153b8087afcdd914         Pod                      spec.containers{compute}                Normal    Started              kubelet, ospha5.cloud.lab.eng.bos.redhat.com        Started container
5s          13s          5         virt-launcher-testvm-ksr4b.153b80883f24df55         Pod                      spec.containers{compute}                Warning   Unhealthy            kubelet, ospha5.cloud.lab.eng.bos.redhat.com        Readiness probe failed: cat: /tmp/healthy: No such file or directory

6m        6m        1         virt-launcher-testvm-mmv5f.153b80317ba615f9   Pod                                               Normal    Scheduled   default-scheduler                              Successfully assigned virt-launcher-testvm-mmv5f to ospha2.cloud.lab.eng.bos.redhat.com
6m        6m        1         virt-launcher-testvm-mmv5f.153b803229cdf226   Pod       spec.containers{volumeregistryvolume}   Normal    Pulled      kubelet, ospha2.cloud.lab.eng.bos.redhat.com   Container image "kubevirt/cirros-registry-disk-demo" already present on machine
6m        6m        1         virt-launcher-testvm-mmv5f.153b803231450e6b   Pod       spec.containers{volumeregistryvolume}   Normal    Created     kubelet, ospha2.cloud.lab.eng.bos.redhat.com   Created container
6m        6m        1         virt-launcher-testvm-mmv5f.153b80323e6ac05f   Pod       spec.containers{volumeregistryvolume}   Normal    Started     kubelet, ospha2.cloud.lab.eng.bos.redhat.com   Started container
6m        6m        1         virt-launcher-testvm-mmv5f.153b80323e99c774   Pod       spec.containers{compute}                Normal    Pulling     kubelet, ospha2.cloud.lab.eng.bos.redhat.com   pulling image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/cnv11-tech-preview/virt-launcher:1.1"
6m        6m        1         virt-launcher-testvm-mmv5f.153b80370bd70264   Pod       spec.containers{compute}                Normal    Pulled      kubelet, ospha2.cloud.lab.eng.bos.redhat.com   Successfully pulled image "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/cnv11-tech-preview/virt-launcher:1.1"
6m        6m        1         virt-launcher-testvm-mmv5f.153b803711e0852a   Pod       spec.containers{compute}                Normal    Created     kubelet, ospha2.cloud.lab.eng.bos.redhat.com   Created container
6m        6m        1         virt-launcher-testvm-mmv5f.153b80371f9a05af   Pod       spec.containers{compute}                Normal    Started     kubelet, ospha2.cloud.lab.eng.bos.redhat.com   Started container
5m        6m        5         virt-launcher-testvm-mmv5f.153b8037a1697de9   Pod       spec.containers{compute}                Warning   Unhealthy   kubelet, ospha2.cloud.lab.eng.bos.redhat.com   Readiness probe failed: cat: /tmp/healthy: No such file or directory

50s       50s       1         virt-launcher-testvm-mmv5f.153b807fbc3ea703   Pod       spec.containers{compute}                Normal    Killing   kubelet, ospha2.cloud.lab.eng.bos.redhat.com   Killing container with id docker://compute:Need to kill Pod
47s       47s       1         virt-launcher-testvm-mmv5f.153b808055f47aa1   Pod       spec.containers{volumeregistryvolume}   Normal    Killing   kubelet, ospha2.cloud.lab.eng.bos.redhat.com   Killing container with id docker://volumeregistryvolume:Need to kill Pod
[root@ospha1 ~]#

Version-Release number of selected component (if applicable):

CNV-1.1

How reproducible:
Each time attempted

Steps to Reproduce:
1. Install OCP 3.10 on RHEL 7.5
2. Download/install the kubevirt-manifest.rpm and kubevirt-virtctl.rpm
3. Copy to local directory then modify /usr/share/kubevirt/manifest/release/kubevirt.yaml so that the image locataions are correct.
4. oc create -f kubevirt.yaml
5. wait for all pods to be ready
6. oc greate -f https://raw.githubusercontent.com/kubevirt/demo/master/manifests/vm.yaml
7. virtctl start testvm


Actual results:

Watch VM/pods after 5-6 minutes it gets new id.

Expected results:

VM stays running

Additional info:

Comment 1 David Vossel 2018-06-27 19:14:44 UTC
There's a pr upstream related to this issue now. https://github.com/kubevirt/kubevirt/pull/1239

Comment 2 Fabian Deutsch 2018-06-29 12:25:28 UTC
David, can you please create an upstream issue with mileston v0.6.2

Comment 3 David Vossel 2018-06-29 12:52:02 UTC
https://github.com/kubevirt/kubevirt/issues/1243

Comment 5 Nelly Credi 2018-08-20 07:22:04 UTC
@Gabriel, can you please verify?


Note You need to log in before you can comment on or make changes to this bug.